Transformers: The Definitive Guide

Chapter 8. Embracing the Era of Experience: Transformers for Planning, Reasoning, and Coding

This chapter marks not only a pivotal shift in your exploration of transformers but also a shift in AI from human-data-centric approaches to experience-driven learning, which is highly relevant to how transformers can be used and advanced.

So far, you’ve seen how transformers have adapted to modeling data across diverse domains: from language and time series to images, video, audio, and, as explored in Chapter 7, reinforcement learning (RL). That chapter laid the groundwork for your understanding of this new era. The next frontier in transformers is not just about comprehending what is but about learning what to do: to plan, reason, and create new knowledge through active interaction with the world.

In this chapter, you’ll learn how transformer-based models are engineered to embody these capabilities. You’ll look into various approaches and architectures that enable transformers to excel in planning, reasoning, and coding by learning from experience. This includes exploring how they process streams of continuous interaction; execute actions and observe their grounded consequences, using objectively verifiable rewards to drive self-improvement; and develop new non-human reasoning strategies. You’ll understand how LLMs are incentivized to develop remarkable reasoning capabilities via RL, showcasing their “aha moments” and impressive performance in complex domains like mathematics and competitive ...