Transformers: The Definitive Guide

Preface

When I first started working with transformers in 2019, I was immediately drawn to them. Two years later, I built my own deep learning architecture based on attention, and that same year I taught my first university lecture on vision transformers. What struck me then, and continues to do so today, is just how versatile this architecture really is. Once you understand how transformers work, patterns begin to emerge. You start to see how the same principles apply across images, video, audio, and beyond. The deeper insight is not found in treating each domain as a separate problem, as is often the case in deep learning, but in recognizing how naturally the transformer architecture extends across domains.

As it turns out, it doesn’t matter whether you’re modeling discrete tokens, image patches, or spectrogram frames. The core logic remains the same. You may change the axis of representation, but the underlying architecture stays intact. That’s the elegance of transformers, at least to me, and the reason I decided to write this book: to share my passion for transformers and let you discover that elegance at your own pace.

What also continues to fascinate me is what happens when transformers are used as part of a larger system. Their real power shows up when you combine multiple models into an agent that can reason, act, and iterate on a task. Beyond simply connecting models, my recent focus has been on pairing transformers with test-time compute and reinforcement learning. ...