[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"learning-part-19-multimodal":3,"learning-all":13},{"id":4,"slug":5,"title":6,"description":7,"level":8,"topic":9,"url":10,"content":11,"sortOrder":12},17,"part-19-multimodal","Part 19 · 多模态应用","让 AI 同时\"看得懂图、听得了音、说得出话\"。","ADVANCED","多模态",null,"# Part 17 · 多模态应用\n\n\n---\n\n## Section 1 · 多模态是什么（概念）\n\n**Subsection 1 · 记概念**\n```\n多模态 = 同时处理多种信息形态（文本\u002F图像\u002F音频\u002F视频）\n单模态：只会文本（ChatGPT 文本版）\n多模态：能看图说话、听音辨义、看视频理解（GPT-4o \u002F Gemini \u002F 通义千问VL）\n```\n\n**Subsection 2 · 体验一个多模态 API**\n- 用你注册的通义千问 DashScope，在控制台找到\"多模态\"模型（如 qwen-vl-max），试传一张图片让它描述。\n- ✅ 预期看到：模型能描述图片内容。\n\n**过关**：能说出多模态概念 = Section 1 完成。\n\n---\n\n## Section 2 · CLIP（图文匹配基础）\n\n**Subsection 1 · 概念**\n```\nCLIP = 把图片和文字映射到同一向量空间\n  - 图→向量，文→向量\n  - 语义匹配的图文，向量距离近\n用途：图文搜索、图文匹配、图像分类（零样本）\n```\n\n**Subsection 2 · 跑 CLIP**\ncmd：`pip install open_clip_torch` 或 `pip install transformers`\n新建 `clip_demo.py`（用 transformers）：\n```python\nfrom transformers import CLIPProcessor, CLIPModel\nfrom PIL import Image\nimport requests\n\n# 下载一张示例图（或换成你本地的图片路径）\nimg = Image.open(requests.get(\"https:\u002F\u002Fhttpbin.org\u002Fimage\u002Fjpeg\", stream=True).raw) if False else Image.open(\"your_image.jpg\")\n# 上面那行不行就手动放一张图到项目里，改用：img = Image.open(\"your_image.jpg\")\n```\n- ⚠️ 网络不便时：用本机一张图片文件，路径换成你自己的。\n- 完整代码（有本地图时）：\n```python\nfrom transformers import CLIPProcessor, CLIPModel\nfrom PIL import Image\n\nmodel = CLIPModel.from_pretrained(\"openai\u002Fclip-vit-base-patch32\")\nprocessor = CLIPProcessor.from_pretrained(\"openai\u002Fclip-vit-base-patch32\")\n\nimg = Image.open(\"your_image.jpg\")\ntexts = [\"一只猫\", \"一条狗\", \"一个风景\"]\ninputs = processor(text=texts, images=img, return_tensors=\"pt\", padding=True)\nout = model(**inputs)\nprobs = out.logits_per_image.softmax(dim=1)\nfor t, p in zip(texts, probs[0].tolist()):\n    print(f\"{t}: {p:.2%}\")\n```\n- ✅ 预期看到：和图片最匹配的文本概率最高。\n\n**过关**：CLIP 跑通 = Section 2 完成。\n\n---\n\n## Section 3 · 图像理解（BLIP \u002F 现成多模态模型）\n\n**Subsection 1 · 用现成模型看图说话**\n用 transformers 的 image-to-text pipeline：\n```python\nfrom transformers import pipeline\n\ncaption = pipeline(\"image-to-text\", model=\"Salesforce\u002Fblip-image-captioning-base\")\nresult = caption(\"your_image.jpg\")\nprint(\"描述:\", result[0][\"generated_text\"])\n```\n- ✅ 预期看到：模型用一句话描述图片。\n- 💡 这就是\"看图说话\"的底层。\n\n**过关**：图像理解跑通 = Section 3 完成。\n\n---\n\n## Section 4 · 图像生成概念（扩散模型）\n\n**Subsection 1 · 记概念**\n```\n扩散模型（Diffusion）= 现在的图像生成主流\n思路：先学\"把图加噪到纯噪声\"，再学\"从纯噪声一步步去噪还原出图\"\n你给它一句提示词 → 它从随机噪声\"去噪\"出你想要的图\n代表：DALL-E、Midjourney、Stable Diffusion\n```\n\n**Subsection 2 · 看原理视频**\n- B 站搜 `扩散模型 原理 通俗`，看 20 分钟（看懂\"加噪\u002F去噪\"即可）。\n\n**过关**：能说扩散模型大致思路 = Section 4 完成。\n\n---\n\n## Section 5 · 跑 Stable Diffusion（本地生成图）\n\n**Subsection 1 · 装 diffusers**\ncmd：`pip install diffusers accelerate`\n\n**Subsection 2 · 生成第一张图**\n新建 `sd_demo.py`：\n```python\nfrom diffusers import StableDiffusionPipeline\nimport torch\n\npipe = StableDiffusionPipeline.from_pretrained(\n    \"runwayml\u002Fstable-diffusion-v1-5\", torch_dtype=torch.float16\n).to(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\nprompt = \"a cute cat reading a book, studio lighting\"\nimg = pipe(prompt).images[0]\nimg.save(\"cat.png\")\nprint(\"已生成 cat.png\")\n```\n- ✅ 预期看到：`cat.png` 里是一只读书的猫。\n- ❌ 显存不够：换 CPU + 小步数，或改 `model_id=\"stabilityai\u002Fsd-turbo\"`（更快）。\n- 💡 中文提示词效果一般，英文提示词更稳。\n\n**过关**：能生成图 = Section 5 完成。\n\n---\n\n## Section 6 · 控制生成（提示词技巧）\n\n**Subsection 1 · 记提示词结构**\n```\n好提示词 = 主体 + 动作 + 环境 + 光线 + 风格 + 画质词\n例：a girl walking in a rainy street, neon lights, cinematic, 4k, masterpiece\n负向提示词：not blurry, no text, no watermark（排除不想要的东西）\n```\n\n**Subsection 2 · 生成 5 张对比**\n- 用同一主体、不同风格词，生成 5 张，看差异。\n- ✅ 预期看到：风格词真的改变画面。\n\n**过关**：会写提示词并对比 = Section 6 完成。\n\n---\n\n## Section 7 · ControlNet（精确控制构图）\n\n**Subsection 1 · 概念**\n```\nControlNet = 给生成加\"控制条件\"：姿势\u002F边缘\u002F深度\u002F线稿\n例：给一张火柴人姿势 → 生成符合该姿势的人物\n用途：电商图、设计稿、一致性构图\n```\n\n**Subsection 2 · 看资料**\n- B 站搜 `ControlNet 教程` 看 20 分钟（了解能控制哪些维度即可，实操可选）。\n\n**过关**：能说 ControlNet 解决什么 = Section 7 完成。\n\n---\n\n## Section 8 · 语音合成 TTS\n\n**Subsection 1 · 跑 TTS**\ncmd：`pip install TTS`（或 `edge-tts`，更轻）\n用 edge-tts（免费微软语音）：\n```\npip install edge-tts\n```\n命令行：\n```\nedge-tts --voice zh-CN-XiaoxiaoNeural --text \"你好，我是你的AI助手。\" --write-media hello.mp3\n```\n- ✅ 预期看到：生成 hello.mp3，能播放中文语音。\n- 💡 TTS = 文字转语音，是做\"AI 语音助手\u002F口播视频\"的基础。\n\n**过关**：能生成语音 = Section 8 完成。\n\n---\n\n## Section 9 · 多模态 API 应用（通义\u002F豆包）\n\n**Subsection 1 · 用 DashScope 多模态**\n- 打开 DashScope 控制台 → 找到 qwen-vl 或图像理解 API 文档。\n- 用 Python 调 qwen-vl，传一张图 + 一个问题（\"图里有什么\"）。\n- ✅ 预期看到：API 返回图片描述。\n- 💡 对比 Section 3 本地 BLIP：API 效果更强但花钱；本地省但弱。按需选。\n\n**过关**：多模态 API 跑通 = Section 9 完成。\n\n---\n\n## Section 10 · 综合 demo：识图 + 问答 + 语音\n\n**Subsection 1 · 组装**\n新建 `multimodal_demo.py`：输入一张图 → 用多模态模型描述 → 再用 TTS 读出来：\n```python\n# 1. 图 → 文本（用 DashScope 或本地 BLIP，Section 3\u002F9 的代码）\ndescription = \"（图片描述结果）\"\n\n# 2. 文本 → 语音（edge-tts）\nimport subprocess\nsubprocess.run([\"edge-tts\", \"--voice\", \"zh-CN-XiaoxiaoNeural\", \"--text\", description, \"--write-media\", \"out.mp3\"])\nprint(\"已生成语音 out.mp3\")\n```\n- ✅ 预期看到：一张图 → 一段文字 → 一段语音，全链路通。\n\n**过关**：端到端多模态 demo 跑通 = Section 10 完成。\n\n---\n\n## Section 11 · 综合项目：AI 看图助手（起）\n\n**Subsection 1 · 定需求 + 搭架子**\n- 项目名 `ai_vision_helper`：上传图片 → 多模态模型回答关于图的问题 → Web 界面展示。\n- 用 FastAPI 搭后端（复用Part 5 的服务器套路）+ 简单 HTML 上传页。\n\n**过关**：架子搭好 = Section 11 完成。\n\n---\n\n## Section 12 · 综合项目：接入模型 + 流式\n\n**Subsection 1 · 接多模态 + 流式回答**\n- 接入 qwen-vl 或 BLIP，支持用户问图。\n- 回答用流式（SSE）返回，前端打字机效果。\n\n**过关**：能问图回答 = Section 12 完成。\n\n---\n\n## Section 13 · 综合项目：加 TTS 朗读\n\n**Subsection 1 · 加语音**\n- 在回答旁加\"朗读\"按钮，点一下用 edge-tts 生成并播放语音。\n\n**过关**：识图 + 朗读全通 = Section 13 完成。\n\n---\n\n## Section 14 · 综合项目：打磨 + 部署\n\n**Subsection 1 · 打磨 + README**\n- 界面美化、错误处理、README（架构图\u002F启动步骤\u002F演示截图）。\n- 部署到服务器或本地可访问。\n\n**过关**：项目可交付 = Section 14 完成。\n\n---\n\n## Section 15 · 进阶：视频理解（可选）\n\n**Subsection 1 · 概念 + 试**\n- 了解视频理解 = 抽帧 + 多模态逐帧理解 + 时序整合。\n- 用 qwen-vl 的\"视频理解\"能力（DashScope 文档有）试一次短视频问答（可选）。\n\n**过关**：了解视频理解思路 = Section 15 完成。\n\n---\n\n## Section 16 · 进阶：语音识别 ASR（可选）\n\n**Subsection 1 · 试 ASR**\n- 用 edge-tts 生成一段语音 → 再用 whisper（`pip install openai-whisper`）转回文字。\n- ✅ 预期看到：语音 → 文字（语音识别闭环）。\n\n**过关**：ASR 跑通 = Section 16 完成。\n\n---\n\n## Section 17 · 进阶：图生图 \u002F 风格迁移（可选）\n\n**Subsection 1 · 用 SD 做图生图**\n- diffusers 里用 `img2img`：给一张图 + 提示词 → 生成\"同构图换风格\"的新图。\n- ✅ 预期看到：图风格改变。\n\n**过关**：图生图跑通 = Section 17 完成。\n\n---\n\n## Section 18 · 多模态产品思路\n\n**Subsection 1 · 记落地场景**\n```\n电商     商品图生成\u002F描述\u002F换背景\n教育     拍照搜题\u002FPPT 转讲解\n办公     表格\u002F发票 OCR → 结构化\n医疗     医学影像辅助解读（仅辅助）\n内容     AI 配音\u002F数字人\u002F口播视频\n```\n- 结合你的情况想一个可做的方向，写进笔记。\n\n**过关**：能说出落地场景 = Section 18 完成。\n\n---\n\n## Section 19 · 面试题\n\n**Subsection 1 · 练习**\n1. 什么是多模态？和单模态区别？\n2. CLIP 是怎么对齐图文？\n3. 扩散模型生成图像的基本思路？\n4. Stable Diffusion 和 GAN 的区别（了解即可）？\n5. 你做过多模态什么项目？\n- 先自己答，再让 DeepSeek 补充。\n\n**过关**：5 题能答 = Section 19 完成。\n\n---\n\n## Section 20 · 站 17 验收\n\n**勾选**\n- [ ] 能说多模态概念\n- [ ] CLIP 图文匹配跑通\n- [ ] 图像理解（BLIP 或 API）跑通\n- [ ] Stable Diffusion 生成图跑通\n- [ ] 能写好生成提示词\n- [ ] 了解 ControlNet\n- [ ] TTS 生成语音跑通\n- [ ] 多模态 API 跑通\n- [ ] 识图+问答+语音端到端 demo 跑通\n- [ ] 综合项目 ai_vision_helper 完成\n- [ ] 5 道面试题能答\n\n**写 300 字Part 17 总结**：你的多模态项目能干什么、用了哪些模型。\n\n**全勾选 = Part 19 通过** → 进入下一站 Part 20 · 项目实战 + 求职备战）。\n\n\n\n",19,[14,21,27,33,40,46,52,58,63,69,75,81,87,93,99,105,111,116,122,123,129],{"id":15,"slug":16,"title":17,"description":18,"level":19,"topic":20,"url":10,"content":10,"sortOrder":15},1,"part-1-llm-basics","Part 1 · 大模型基础认知","小白零基础友好。这一站不写代码，只建立\"大模型到底是什么\"的骨架。","BEGINNER","大模型认知",{"id":22,"slug":23,"title":24,"description":25,"level":19,"topic":26,"url":10,"content":10,"sortOrder":22},2,"part-2-llm-principles","Part 2 · 大模型原理 · 入门实操","这一站不卷数学，只把\"理解模型所需的知识\"讲透。；站 2A 解决\"理解模型\"，站 2B 解决\"面试能答\"。；这一层是\"有余力再做\"，不做也不影响你进入站 3。","大模型原理",{"id":28,"slug":29,"title":30,"description":31,"level":19,"topic":26,"url":10,"content":10,"sortOrder":32},100,"part-3-llm-principles-deep","Part 3 · 大模型原理 · 面试深入","数学基础、机器学习、神经网络、词向量、思维链等面试必考原理",3,{"id":34,"slug":35,"title":36,"description":37,"level":38,"topic":26,"url":10,"content":10,"sortOrder":39},101,"part-4-llm-principles-advanced","Part 4 · 大模型原理 · 进阶可选","从零写迷你模型、RLHF、PPO、分布式训练实操等进阶内容","INTERMEDIATE",4,{"id":32,"slug":41,"title":42,"description":43,"level":19,"topic":44,"url":10,"content":10,"sortOrder":45},"part-5-prompt-engineering","Part 5 · Prompt 提示词工程","从这里开始，你每天都要真的调大模型 API。","提示工程",5,{"id":39,"slug":47,"title":48,"description":49,"level":19,"topic":50,"url":10,"content":10,"sortOrder":51},"part-6-llm-api","Part 6 · 大模型 API 调用","站 3 你已经调通了 API。这一站把\"调用\"做成专业水准：","API 工程",6,{"id":45,"slug":53,"title":54,"description":55,"level":38,"topic":56,"url":10,"content":10,"sortOrder":57},"part-7-rag","Part 7 · RAG 检索增强生成","这一站是\"能落地\"的关键：让模型回答**你的私有知识**。","RAG",7,{"id":51,"slug":59,"title":60,"description":61,"level":38,"topic":56,"url":10,"content":10,"sortOrder":62},"part-8-rag-optimization","Part 8 · RAG 优化与评估","站 5 你做出了\"能跑\"的 RAG。这一站把它做到\"能打\"：",8,{"id":57,"slug":64,"title":65,"description":66,"level":38,"topic":67,"url":10,"content":10,"sortOrder":68},"part-9-langchain","Part 9 · LangChain 框架","目标：用 LangChain 把\"模型、提示词、检索、记忆、工具\"串成可复用的链和 Agent。","LangChain",9,{"id":62,"slug":70,"title":71,"description":72,"level":38,"topic":73,"url":10,"content":10,"sortOrder":74},"part-10-agent","Part 10 · Agent 基础","让模型从\"会聊天\"变成\"能干活\"：自己拆任务、调工具、看结果、再行动。","Agent",10,{"id":68,"slug":76,"title":77,"description":78,"level":8,"topic":79,"url":10,"content":10,"sortOrder":80},"part-11-multi-agent","Part 11 · 多 Agent 与 Agent IDE","从\"单个 Agent\"升级到\"多 Agent 协作\"，并掌握 AutoGen \u002F LangGraph \u002F GPTs \u002F Coze \u002F Dify 五个工具。","多 Agent",11,{"id":74,"slug":82,"title":83,"description":84,"level":8,"topic":85,"url":10,"content":10,"sortOrder":86},"part-12-llamaindex","Part 12 · LlamaIndex","站 5~7 你用了 LangChain 做 RAG。LlamaIndex 是另一条专攻\"数据 + LLM\"的路线。","LlamaIndex",12,{"id":80,"slug":88,"title":89,"description":90,"level":8,"topic":91,"url":10,"content":10,"sortOrder":92},"part-13-transformer","Part 13 · Transformer 深入","站 2A 建立了直觉，这一站把手写代码跑通——从\"懂概念\"到\"写得出\"。","Transformer",13,{"id":86,"slug":94,"title":95,"description":96,"level":8,"topic":97,"url":10,"content":10,"sortOrder":98},"part-14-open-models","Part 14 · 开源模型与私有化部署","目标：把主流开源模型在本地\u002F服务器跑通，理解服务化工程（显存、量化、并发）。","模型部署",14,{"id":92,"slug":100,"title":101,"description":102,"level":8,"topic":103,"url":10,"content":10,"sortOrder":104},"part-15-finetuning","Part 15 · 模型微调 Fine-Tuning","目标：理解\"什么时候必须微调、怎么选基座、怎么备数据\"，并完整跑通一次业务微调。","微调",15,{"id":98,"slug":106,"title":107,"description":108,"level":8,"topic":109,"url":10,"content":10,"sortOrder":110},"part-16-peft-lora","Part 16 · PEFT 参数高效微调","重点吃透 **LoRA**：用极少量可训练参数微调大模型（消费级显卡就能跑）。","PEFT\u002FLoRA",16,{"id":104,"slug":112,"title":113,"description":114,"level":8,"topic":115,"url":10,"content":10,"sortOrder":4},"part-17-quantization","Part 17 · 模型量化","目标：理解\"为什么量化、主流算法各自思路\"，能动手量化并评估效果。","量化",{"id":110,"slug":117,"title":118,"description":119,"level":8,"topic":120,"url":10,"content":10,"sortOrder":121},"part-18-data-evaluation","Part 18 · 训练数据与模型评估","数据决定模型上限，评估决定你\"能不能交付\"。","数据与评估",18,{"id":4,"slug":5,"title":6,"description":7,"level":8,"topic":9,"url":10,"content":10,"sortOrder":12},{"id":121,"slug":124,"title":125,"description":126,"level":8,"topic":127,"url":10,"content":10,"sortOrder":128},"part-20-projects-career","Part 20 · 项目实战 + 求职备战","把 0→1 到 1→100 学到的全部收敛成**能展示、能讲、能面试**的项目。","项目与求职",20,{"id":130,"slug":131,"title":132,"description":133,"level":8,"topic":134,"url":10,"content":10,"sortOrder":135},102,"extra-2026-ai-tech","额外篇 · 2026 年 AI 应用开发必学的 5 项成熟技术","从近半年爆发的技术里，挑出已经过了尝鲜期、能真正用在项目里的 5 项：推理模型、MCP、多模态、GraphRAG、AI 编程工具链，每项都给到能跑通的代码。","AI 工程实践",21]