Generative Deep Learning, 2nd Edition

Chapter 9. Transformers

Chapter Goals

In this chapter you will:

• Learn about the origins of GPT, a powerful decoder Transformer model for text generation.

• Learn conceptually how an attention mechanism mimics our way of attaching more importance to some words in a sentence than others.

• Delve into how the attention mechanism works from first principles, including how queries, keys, and values are created and manipulated.

• See the importance of causal masking for text generation tasks.

• Understand how attention heads can be grouped into a multihead attention layer.

• See how multihead attention layers form one part of a Transformer block that also includes layer normalization and skip connections.

• Create positional encodings that capture the position of each token as well as the word token embedding.

• Build a GPT model in Keras to generate the text contained in wine reviews.

• Analyze the output from the GPT model, including interrogating the attention scores to inspect where the model is looking.

• Learn about the different types of Transformers, including examples of the types of tasks that can be tackled by each and descriptions of the most famous state-of-the-art implementations.

• Understand how encoder-decoder architectures work, like Google’s T5 model.

• Explore the training process behind OpenAI’s ChatGPT.

We saw in Chapter 5 how we can build generative models on text data using recurrent neural networks (RNNs), such as LSTMs and GRUs. These autoregressive models ...