Transformers: The Definitive Guide

Chapter 4. Transformers for Image Generation

A new era of photorealistic image creation has started with the development of text-to-image (T2I) generative models such as DALL·E, Imagen, Midjourney and Stable Diffusion. These models can generate highly detailed and contextually accurate images and artwork from textual inputs, enabling artists, designers, and even non-creative individuals to bring their ideas to life with unprecedented ease.

The core building block of these models is diffusion, a process in physics that describes how molecules move from high-concentration to low-concentration areas. In the context of image generation, diffusion models leverage this principle to progressively transform random noise into coherent and detailed images through a series of iterative refinements.

This diffusion-based approach has become the backbone of SOTA models, replacing earlier methods such as Generative Adversarial Networks (GANs),¹ which previously dominated image generation. This shift occurred because diffusion models have proven to be more effective in producing high-quality and diverse outputs. Notably, while the original DALL·E model was autoregressive, DALL·E 2 and DALL·E 3 have since adopted diffusion-based methods as their approach.

Given the current research interest in diffusion models, this chapter will focus on diffusion models such as the first diffusion transformer (DiT), PIXART-α, PixArt-Σ, and DiffiT for image-generation tasks. Additionally, I’ll focus on open source ...