Transformers: The Definitive Guide

Chapter 1. From First Principles to State-of-the-Art Transformers

Since its introduction in 2017, the transformer architecture¹ has revolutionized the field of natural language processing (NLP), marking a paradigm shift toward models capable of natural language understanding (NLU). This shift was possible because transformers process sequential data in parallel, enabling a deeper and more contextual understanding of language than was achievable with previous sequential models, like long short-term memory (LSTM) networks.

In recent years, transformers have evolved to impact a wide array of domains, including computer vision, speech recognition, reinforcement learning, and mathematical operations, moving beyond their initial usage within NLP. Their adaptability has led to significant advancements in machine translation, allowing for context-aware translations, and in scientific research, notably in predicting protein structures with remarkable accuracy.

Among the most exciting developments are reasoning models, which are advanced large language models (LLMs) trained with reinforcement learning to perform complex, multi-step reasoning. They generate internal chains of thought before answering, which is inspired by the human thought process. This technique first solves intermediate steps before getting to the final answer.

I assume in this book that you have at least some familiarity with the transformer architecture. Perhaps you’re read the book Natural Language Processing with Transformers ...