[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"learning-part-13-transformer":3,"learning-all":13},{"id":4,"slug":5,"title":6,"description":7,"level":8,"topic":9,"url":10,"content":11,"sortOrder":12},11,"part-13-transformer","Part 13 · Transformer 深入","站 2A 建立了直觉，这一站把手写代码跑通——从\"懂概念\"到\"写得出\"。","ADVANCED","Transformer",null,"# Part 11 · Transformer 深入\n\n\n---\n\n## Section 1 · 自注意力再复习（从\"看\"到\"讲\"）\n\n**Subsection 1 · 看完整版**\n- 打开 https:\u002F\u002Fwww.bilibili.com\u002Fvideo\u002FBV1v3411r78R （李宏毅：自注意力机制和 Transformer 详解）看 **30:00~60:00**（注意力 + 多头部分）。\n- 重点盯：Q、K、V 是什么、注意力分数怎么算。\n\n**Subsection 2 · 用一句话给\"没学过的人\"讲**\n写下来（照抄，理解着抄）：\n> 每个词都生成三个向量：Query（我要找谁）、Key（我是什么）、Value（我的内容）。\n> 分数 = 我的 Q 和别人的 K 做点积 → 谁和我最搭，我就多关注谁的 V。\n> 最后所有 V 加权平均 = 我这个词融合了全句信息的\"新表示\"。\n\n**过关**：能对着 Q\u002FK\u002FV 讲一遍 = Section 1 完成。\n\n---\n\n## Section 2 · 用代码算一次注意力（不靠库）\n\n**Subsection 1 · 跑手动实现**\n新建 `attn_manual.py`：\n```python\nimport torch\nimport torch.nn.functional as F\n\n# 3 个词，每个 4 维\nX = torch.tensor([\n    [1.0, 0.0, 0.0, 0.0],  # 词A \"我\"\n    [0.0, 1.0, 0.0, 0.0],  # 词B \"爱\"\n    [0.0, 0.0, 1.0, 0.0],  # 词C \"你\"\n])\n\n# 简化：直接用 X 当 Q、K、V（真实模型会用可学习权重变换）\nQ = K = V = X\n\n# 1. 算注意力分数：Q 和所有 K 的点积\nscores = torch.mm(Q, K.T)          # 3x3\nprint(\"注意力分数:\\n\", scores)\n\n# 2. 归一化成权重（softmax，按行）\nweights = F.softmax(scores, dim=-1)\nprint(\"\\n注意力权重:\\n\", weights)\n\n# 3. 加权求和 V\nout = torch.mm(weights, V)\nprint(\"\\n输出（每个词融合了全句信息）:\\n\", out)\n```\n- ✅ 预期看到：一个 3x3 的分数矩阵、权重矩阵、输出矩阵。\n- 💡 这就是注意力的**全部内核**，真实实现只是加了三组可学习权重 Wq\u002FWk\u002FWv 和多头。\n\n**过关**：能跑通并解释每行干嘛 = Section 2 完成。\n\n---\n\n## Section 3 · 多头注意力（为什么分\"头\"）\n\n**Subsection 1 · 概念**\n```\n多头 = 用多组 Wq\u002FWk\u002FWv 同时算多次注意力，每组关注不同关系\n例：头1 关注\"语法搭配\"，头2 关注\"指代关系\"，头3 关注\"数字大小\"\n最后把头拼起来 → 模型看得更全面\n```\n\n**Subsection 2 · 跑多头**\n新建 `multi_head.py`：\n```python\nimport torch\nimport torch.nn as nn\n\nclass MultiHeadAttention(nn.Module):\n    def __init__(self, d_model=4, n_heads=2):\n        super().__init__()\n        self.n_heads = n_heads\n        self.head_dim = d_model \u002F\u002F n_heads\n        # 每组头用自己的权重\n        self.wq = nn.Linear(d_model, d_model)\n        self.wk = nn.Linear(d_model, d_model)\n        self.wv = nn.Linear(d_model, d_model)\n        self.out = nn.Linear(d_model, d_model)\n\n    def forward(self, x):\n        B, T, D = x.shape\n        Q = self.wq(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2)\n        K = self.wk(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2)\n        V = self.wv(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2)\n        scores = Q @ K.transpose(-2, -1) \u002F (self.head_dim ** 0.5)\n        weights = torch.softmax(scores, dim=-1)\n        out = (weights @ V).transpose(1, 2).reshape(B, T, D)\n        return self.out(out)\n\nx = torch.randn(1, 3, 4)   # 1句话3个词，4维\nprint(MultiHeadAttention()(x).shape)   # torch.Size([1, 3, 4])\n```\n- ✅ 预期看到：`torch.Size([1, 3, 4])`（输入输出维度一致）。\n- 💡 这已经是真实多头注意力的骨架，跑通它你就\"会写\"了。\n\n**过关**：多头代码跑通 = Section 3 完成。\n\n---\n\n## Section 4 · 位置编码（模型怎么知道顺序）\n\n**Subsection 1 · 概念**\n```\n注意力不分前后顺序（\"我打你\"和\"你打我\"权重一样）→ 需要位置编码\n位置编码 = 给每个词的位置加一组特殊向量，让模型知道\"谁在前谁在后\"\n经典做法：sin\u002Fcos 公式（原论文）或 可学习的位置向量（GPT 用）\n```\n\n**Subsection 2 · 跑位置编码**\n新建 `pos_enc.py`：\n```python\nimport torch\nimport math\n\ndef position_encoding(seq_len=4, d_model=8):\n    pe = torch.zeros(seq_len, d_model)\n    for pos in range(seq_len):\n        for i in range(0, d_model, 2):\n            pe[pos, i] = math.sin(pos \u002F (10000 ** (i \u002F d_model)))\n            pe[pos, i + 1] = math.cos(pos \u002F (10000 ** (i \u002F d_model)))\n    return pe\n\npe = position_encoding()\nprint(\"位置0的编码:\", pe[0])\nprint(\"位置1的编码:\", pe[1])\nprint(\"不同位置编码不同 → 模型能区分顺序\")\n```\n- ✅ 预期看到：每个位置的编码向量不一样。\n\n**过关**：能跑通并说\"位置编码解决什么\" = Section 4 完成。\n\n---\n\n## Section 5 · Encoder vs Decoder（两种模型家族）\n\n**Subsection 1 · 记区别**\n```\nEncoder-only   双向看全句（BERT）→ 适合理解\u002F分类\u002F嵌入\nDecoder-only   只往前看（GPT）  → 适合生成（所有现代 LLM 都用它）\nEncoder-Decoder 编码+解码（T5\u002F机器翻译）→ 复杂生成任务\n```\n- 💡 你用的 GPT 系列（含 DeepSeek）都是 Decoder-only。\n\n**Subsection 2 · 掩码自注意力（为什么\"不能看未来\"）**\n新建 `mask_attn.py`：\n```python\nimport torch\n\nscores = torch.rand(4, 4)   # 4个词的注意力分数\n# 掩码：只允许看自己和之前（上三角置为 -inf）\nmask = torch.triu(torch.full((4, 4), float(\"-inf\")), diagonal=1)\nmasked = scores + mask\nprint(\"掩码后（下三角可见，上三角=无效）:\\n\", torch.round(masked, decimals=2))\n```\n- ✅ 预期看到：右上三角全是 -inf。\n- 💡 Decoder 生成时就是\"一次只能看前面的词\"，保证生成不\"作弊\"。\n\n**过关**：能说 Encoder\u002FDecoder 区别 + 掩码作用 = Section 5 完成。\n\n---\n\n## Section 6 · 解码策略（模型怎么选下一个词）\n\n**Subsection 1 · 跑三种解码**\n新建 `decoding.py`：\n```python\nimport torch\nimport torch.nn.functional as F\n\n# 假设模型给出 5 个候选词的分数\nlogits = torch.tensor([2.0, 1.5, 0.1, -1.0, 3.5])\n\n# 1. 贪心：直接取分数最高的\nprint(\"贪心:\", logits.argmax().item())\n\n# 2. 采样（temperature）：分数缩放后随机抽\nprobs = F.softmax(logits \u002F 0.8, dim=-1)\nsample = torch.multinomial(probs, 1).item()\nprint(\"采样(温度0.8):\", sample, \"| 概率分布:\", [round(p, 2) for p in probs])\n\n# 3. Top-P（核采样）：只从累计概率到 p 的词里抽\ndef top_p_sampling(logits, p=0.9):\n    probs = F.softmax(logits, dim=-1)\n    sorted_p, idx = torch.sort(probs, descending=True)\n    cum = torch.cumsum(sorted_p, dim=-1)\n    mask = cum - sorted_p > p\n    sorted_p[mask] = 0\n    probs = sorted_p \u002F sorted_p.sum()\n    return torch.multinomial(probs, 1)\nprint(\"Top-P 采样:\", top_p_sampling(logits).item())\n```\n- ✅ 预期看到：三种方法分别选出词。\n- 💡 面试常问：**贪心稳但死板，采样多样但可能跑偏；生产常用 temperature + top_p 组合**。\n\n**过关**：能跑通并说出三方法取舍 = Section 6 完成。\n\n---\n\n## Section 7 · 手写完整 Self-Attention 层\n\n**Subsection 1 · 跑完整代码**\n新建 `my_attention.py`（把 Section 2\u002F3 合并成规范实现）：\n```python\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nclass SelfAttention(nn.Module):\n    def __init__(self, d_model=8):\n        super().__init__()\n        self.wq = nn.Linear(d_model, d_model)\n        self.wk = nn.Linear(d_model, d_model)\n        self.wv = nn.Linear(d_model, d_model)\n\n    def forward(self, x):\n        Q, K, V = self.wq(x), self.wk(x), self.wv(x)\n        d = Q.shape[-1]\n        scores = torch.mm(Q, K.T) \u002F (d ** 0.5)\n        weights = F.softmax(scores, dim=-1)\n        return torch.mm(weights, V)\n\nx = torch.randn(4, 8)          # 4个词，8维\nout = SelfAttention()(x)\nprint(\"输出形状:\", out.shape)   # torch.Size([4, 8])\n```\n- ✅ 预期看到：`torch.Size([4, 8])`。\n\n**过关**：能独立写出（或对照默写）这个 SelfAttention = Section 7 完成。\n\n---\n\n## Section 8 · 手写一个迷你 GPT（Decoder 骨架）\n\n**Subsection 1 · 跑完整迷你 GPT**\n新建 `mini_gpt.py`：\n```python\nimport torch\nimport torch.nn as nn\n\nclass MiniGPT(nn.Module):\n    def __init__(self, vocab=100, d_model=32, n_heads=4, n_layers=2):\n        super().__init__()\n        self.embed = nn.Embedding(vocab, d_model)\n        self.pos = nn.Parameter(torch.randn(1, 50, d_model))\n        self.blocks = nn.ModuleList([\n            nn.TransformerEncoderLayer(d_model, n_heads, dim_feedforward=128, batch_first=True)\n            for _ in range(n_layers)\n        ])\n        self.head = nn.Linear(d_model, vocab)\n\n    def forward(self, x):\n        x = self.embed(x) + self.pos[:, :x.shape[1]]\n        for b in self.blocks:\n            x = b(x)\n        return self.head(x)   # 每个位置预测下一个词的分数\n\nmodel = MiniGPT()\nx = torch.randint(0, 100, (2, 10))     # 2句话，每句10个词\nlogits = model(x)\nprint(\"输出形状:\", logits.shape)         # torch.Size([2, 10, 100])\nprint(\"第一句第一个位置预测的词表分数前3:\", logits[0, 0, :3].detach().numpy())\n```\n- ✅ 预期看到：`torch.Size([2, 10, 100])`（每句话每位置给词表打分）。\n- 💡 这就是 GPT 的极简骨架：**嵌入 + 位置 + N 层 Transformer + 预测词表分数**。真实的 GPT 只是把它放大 + 用海量数据训练。\n\n**过关**：迷你 GPT 跑通 = Section 8 完成。\n\n---\n\n## Section 9 · 用 HuggingFace 做同样的生成（对照）\n\n**Subsection 1 · 跑 HF 生成**\n新建 `hf_generate.py`：\n```python\nfrom transformers import pipeline\n\n# 加载一个很小的中文生成模型\ngen = pipeline(\"text-generation\", model=\"uer\u002Fgpt2-chinese-cluecorpussmall\")\n\nresult = gen(\"今天天气\", max_new_tokens=20, do_sample=True, temperature=0.8)\nprint(result[0][\"generated_text\"])\n```\n- ✅ 预期看到：模型续写出一小段中文。\n- 💡 对比 Section 8：HF 把\"嵌入+层+head+采样\"全封装好了。你现在明白它内部在干嘛了。\n\n**过关**：能跑通并说\"HF 内部就是我们手写的东西\" = Section 9 完成。\n\n---\n\n## Section 10 · 站 11 验收\n\n**勾选**\n- [ ] 能对着 Q\u002FK\u002FV 讲一遍注意力\n- [ ] 手动注意力代码跑通（Section 2）\n- [ ] 多头注意力跑通（Section 3）\n- [ ] 位置编码跑通，说清作用（Section 4）\n- [ ] 能说 Encoder\u002FDecoder 区别 + 掩码（Section 5）\n- [ ] 三种解码策略跑通并说取舍（Section 6）\n- [ ] 手写 SelfAttention 跑通（Section 7）\n- [ ] 手写迷你 GPT 跑通（Section 8）\n- [ ] HF 生成对照跑通（Section 9）\n\n**写 300 字Part 11 总结**：从概念到代码，你现在对 Transformer 的理解。\n\n**全勾选 = Part 13 通过** → 进入下一站 Part 14 · 开源模型与私有化部署，20 天）。\n\n\n\n",13,[14,21,27,33,40,46,52,58,63,69,75,80,86,87,93,99,105,111,117,123,129],{"id":15,"slug":16,"title":17,"description":18,"level":19,"topic":20,"url":10,"content":10,"sortOrder":15},1,"part-1-llm-basics","Part 1 · 大模型基础认知","小白零基础友好。这一站不写代码，只建立\"大模型到底是什么\"的骨架。","BEGINNER","大模型认知",{"id":22,"slug":23,"title":24,"description":25,"level":19,"topic":26,"url":10,"content":10,"sortOrder":22},2,"part-2-llm-principles","Part 2 · 大模型原理 · 入门实操","这一站不卷数学，只把\"理解模型所需的知识\"讲透。；站 2A 解决\"理解模型\"，站 2B 解决\"面试能答\"。；这一层是\"有余力再做\"，不做也不影响你进入站 3。","大模型原理",{"id":28,"slug":29,"title":30,"description":31,"level":19,"topic":26,"url":10,"content":10,"sortOrder":32},100,"part-3-llm-principles-deep","Part 3 · 大模型原理 · 面试深入","数学基础、机器学习、神经网络、词向量、思维链等面试必考原理",3,{"id":34,"slug":35,"title":36,"description":37,"level":38,"topic":26,"url":10,"content":10,"sortOrder":39},101,"part-4-llm-principles-advanced","Part 4 · 大模型原理 · 进阶可选","从零写迷你模型、RLHF、PPO、分布式训练实操等进阶内容","INTERMEDIATE",4,{"id":32,"slug":41,"title":42,"description":43,"level":19,"topic":44,"url":10,"content":10,"sortOrder":45},"part-5-prompt-engineering","Part 5 · Prompt 提示词工程","从这里开始，你每天都要真的调大模型 API。","提示工程",5,{"id":39,"slug":47,"title":48,"description":49,"level":19,"topic":50,"url":10,"content":10,"sortOrder":51},"part-6-llm-api","Part 6 · 大模型 API 调用","站 3 你已经调通了 API。这一站把\"调用\"做成专业水准：","API 工程",6,{"id":45,"slug":53,"title":54,"description":55,"level":38,"topic":56,"url":10,"content":10,"sortOrder":57},"part-7-rag","Part 7 · RAG 检索增强生成","这一站是\"能落地\"的关键：让模型回答**你的私有知识**。","RAG",7,{"id":51,"slug":59,"title":60,"description":61,"level":38,"topic":56,"url":10,"content":10,"sortOrder":62},"part-8-rag-optimization","Part 8 · RAG 优化与评估","站 5 你做出了\"能跑\"的 RAG。这一站把它做到\"能打\"：",8,{"id":57,"slug":64,"title":65,"description":66,"level":38,"topic":67,"url":10,"content":10,"sortOrder":68},"part-9-langchain","Part 9 · LangChain 框架","目标：用 LangChain 把\"模型、提示词、检索、记忆、工具\"串成可复用的链和 Agent。","LangChain",9,{"id":62,"slug":70,"title":71,"description":72,"level":38,"topic":73,"url":10,"content":10,"sortOrder":74},"part-10-agent","Part 10 · Agent 基础","让模型从\"会聊天\"变成\"能干活\"：自己拆任务、调工具、看结果、再行动。","Agent",10,{"id":68,"slug":76,"title":77,"description":78,"level":8,"topic":79,"url":10,"content":10,"sortOrder":4},"part-11-multi-agent","Part 11 · 多 Agent 与 Agent IDE","从\"单个 Agent\"升级到\"多 Agent 协作\"，并掌握 AutoGen \u002F LangGraph \u002F GPTs \u002F Coze \u002F Dify 五个工具。","多 Agent",{"id":74,"slug":81,"title":82,"description":83,"level":8,"topic":84,"url":10,"content":10,"sortOrder":85},"part-12-llamaindex","Part 12 · LlamaIndex","站 5~7 你用了 LangChain 做 RAG。LlamaIndex 是另一条专攻\"数据 + LLM\"的路线。","LlamaIndex",12,{"id":4,"slug":5,"title":6,"description":7,"level":8,"topic":9,"url":10,"content":10,"sortOrder":12},{"id":85,"slug":88,"title":89,"description":90,"level":8,"topic":91,"url":10,"content":10,"sortOrder":92},"part-14-open-models","Part 14 · 开源模型与私有化部署","目标：把主流开源模型在本地\u002F服务器跑通，理解服务化工程（显存、量化、并发）。","模型部署",14,{"id":12,"slug":94,"title":95,"description":96,"level":8,"topic":97,"url":10,"content":10,"sortOrder":98},"part-15-finetuning","Part 15 · 模型微调 Fine-Tuning","目标：理解\"什么时候必须微调、怎么选基座、怎么备数据\"，并完整跑通一次业务微调。","微调",15,{"id":92,"slug":100,"title":101,"description":102,"level":8,"topic":103,"url":10,"content":10,"sortOrder":104},"part-16-peft-lora","Part 16 · PEFT 参数高效微调","重点吃透 **LoRA**：用极少量可训练参数微调大模型（消费级显卡就能跑）。","PEFT\u002FLoRA",16,{"id":98,"slug":106,"title":107,"description":108,"level":8,"topic":109,"url":10,"content":10,"sortOrder":110},"part-17-quantization","Part 17 · 模型量化","目标：理解\"为什么量化、主流算法各自思路\"，能动手量化并评估效果。","量化",17,{"id":104,"slug":112,"title":113,"description":114,"level":8,"topic":115,"url":10,"content":10,"sortOrder":116},"part-18-data-evaluation","Part 18 · 训练数据与模型评估","数据决定模型上限，评估决定你\"能不能交付\"。","数据与评估",18,{"id":110,"slug":118,"title":119,"description":120,"level":8,"topic":121,"url":10,"content":10,"sortOrder":122},"part-19-multimodal","Part 19 · 多模态应用","让 AI 同时\"看得懂图、听得了音、说得出话\"。","多模态",19,{"id":116,"slug":124,"title":125,"description":126,"level":8,"topic":127,"url":10,"content":10,"sortOrder":128},"part-20-projects-career","Part 20 · 项目实战 + 求职备战","把 0→1 到 1→100 学到的全部收敛成**能展示、能讲、能面试**的项目。","项目与求职",20,{"id":130,"slug":131,"title":132,"description":133,"level":8,"topic":134,"url":10,"content":10,"sortOrder":135},102,"extra-2026-ai-tech","额外篇 · 2026 年 AI 应用开发必学的 5 项成熟技术","从近半年爆发的技术里，挑出已经过了尝鲜期、能真正用在项目里的 5 项：推理模型、MCP、多模态、GraphRAG、AI 编程工具链，每项都给到能跑通的代码。","AI 工程实践",21]