Transformers: The Definitive Guide

Chapter 7. Reinforcement Learning Transformers

Reinforcement learning (RL) is a paradigm based on making sequential decisions through feedback from an environment. If you think about it, learning through interaction is not a new concept. In fact, you’ve been familiar with the basic principles of RL all your life. This is because you often learn by interacting with your surroundings without needing an explicit teacher. For example, as a child, you might have touched things or put objects in your mouth to see how your environment reacted. Based on the outcome, such as crying after being scratched by the family cat, you learned what to do next. From such experiences, you developed a “policy” for future actions, like not pulling the cat’s tail again.

This learning process mirrors how an RL agent operates. Consequently, this chapter will leverage your natural understanding of learning from an environment and build on it by discussing key RL concepts. Specifically, I’ll provide an introductory overview of traditional RL algorithms, including distinctions between online and offline learning, model-based and model-free approaches, and on-policy versus off-policy RL.

Once you have a basic understanding of these foundational concepts, you’ll explore recent architectures in RL transformers, such as decision transformer, online decision transformer, and efficient stochastic transformer-based world models.

Getting Started with Reinforcement Learning

Typically, the interaction with the environment ...