March 2026
Intermediate
372 pages
4h 53m
Chinese
Content preview from 变形金刚权威指南 (Chinese Edition)
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
第8章 拥抱 体验时代： 用于规划、 推理和编码的Transformer
本作品已使用人工智能进行翻译。欢迎您提供反馈和意见：translation-feedback@oreilly.com
本书 章不仅标志着您对Transformer探索的关键转折点，也标志着人工智能从以人-数据为中心的方法向体验驱动的学习转变，这与Transformer的应用和进步密切相关。
迄今为止，您已经看到Transformer如何适应不同领域的数据建模：从语言和时间序列到图像、视频、音频，以及第7章探讨的强化学习（RL）。那一章为您理解这个新时代奠定了基础。Transformer的下一个前沿不仅在于理解“是什么”，更在于学习“该做什么”：通过与世界的积极互动来规划、推理并创造新知识。
在本章中，你将学习基于 Transformer 的模型是如何被设计以体现这些能力的。你将深入探讨各种方法和架构，这些方法和架构使 Transformer 能够通过从经验中学习，在规划、推理和编码方面表现出色。这包括探索它们如何处理连续交互流；执行动作并观察其具体后果，利用客观可验证的奖励来推动自我改进；以及开发新的非人类推理策略。 你将理解如何通过强化学习激励LLMs发展出卓越的推理能力，并展示它们在数学和竞赛编程等复杂领域中的“顿悟时刻”和令人印象深刻的表现。你将看到 如何在推理时鼓励、塑造，甚至禁用和限制推理能力，以及如何策略性地给予较小的模型更多时间进行思考和更好的推理——从而甚至能胜过更大的模型。
从人类数据到生活体验
年来，我们一直身处数据时代。如今，我们正步入《设计智能》（麻省理工学院出版社） 的作者、强化学习领域 的巨擘大卫·西尔弗 和理查德·S·萨顿 所描述的“体验时代”。萨顿在强化学习方面的奠基性工作，特别是时序差异学习和Dyna架构等概念，1 塑造了我们对智能系统如何通过试错学习的理解。
若要实现更显著的进步，模型需要一种新的、持续优化的数据来源：其自身的亲身经历。这意味着通过与环境交互进行自主学习，并生成随着模型自身能力增强而不断适应和进化的数据。正是在此处，强化学习重新夺回了其核心地位，为模型提供了“边做边学”的框架，使其能够直接从自身在现实世界中的行动中获得反馈。
对于推理等复杂的认知任务，这种转变蕴含着强大的协同效应：将Transformer生成思维链（CoT）的固有能力2 推理 ，与强化学习提供的迭代优化相结合。虽然初始的CoT能力可能源于人类示例的引导，但“经验时代”通过利用强化学习自主发现并优化推理路径，将这一能力推向了新的高度。这包括那些可能偏离类人思维模式，却能导向客观正确结果的推理路径。 这一关键的反馈循环使Transformer能够将其内部“思维”与环境现实（如数学证明验证器或代码编译器）进行比对，并据此进行优化。
一个关键的例子是的DeepSeek-R1-Zero模型，3 该模型证明，仅通过大规模强化学习即可产生高级推理能力，无需监督式微调。该模型直接基于DeepSeek-V3-Base，利用组相对策略优化（GRPO）框架 进行训练，从而习得了复杂的CoT推理策略。 经过数千次强化学习迭代，该模型展现出持续的自我进化能力，其在 AIME 2024 数据集上的 pass@1 得分（见侧边栏）从 15.6% 提升至 71.0%。AIME 基准数据集包含基于高中数学竞赛建模的数学问题。
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.