book

Generative Deep Learning, 2nd Edition

by David Foster

April 2023

Intermediate to advanced

456 pages

11h 12m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Includes

Includes Quizzes

Objective and ApproachPrerequisitesRoadmapChanges in the Second EditionOther ResourcesConventions Used in This BookCodebaseUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
What Is Generative Modeling?Generative Versus Discriminative ModelingThe Rise of Generative ModelingGenerative Modeling and AIOur First Generative ModelHello World!The Generative Modeling FrameworkRepresentation LearningCore Probability TheoryGenerative Model TaxonomyThe Generative Deep Learning CodebaseCloning the RepositoryUsing DockerRunning on a GPUSummary
Data for Deep LearningDeep Neural NetworksWhat Is a Neural Network?Learning High-Level FeaturesTensorFlow and KerasMultilayer Perceptron (MLP)Preparing the DataBuilding the ModelCompiling the ModelTraining the ModelEvaluating the ModelConvolutional Neural Network (CNN)Convolutional LayersBatch NormalizationDropoutBuilding the CNNTraining and Evaluating the CNNSummary
IntroductionAutoencodersThe Fashion-MNIST DatasetThe Autoencoder ArchitectureThe EncoderThe DecoderJoining the Encoder to the DecoderReconstructing ImagesVisualizing the Latent SpaceGenerating New ImagesVariational AutoencodersThe EncoderThe Loss FunctionTraining the Variational AutoencoderAnalysis of the Variational AutoencoderExploring the Latent SpaceThe CelebA DatasetTraining the Variational AutoencoderAnalysis of the Variational AutoencoderGenerating New FacesLatent Space ArithmeticMorphing Between FacesSummary
IntroductionDeep Convolutional GAN (DCGAN)The Bricks DatasetThe DiscriminatorThe GeneratorTraining the DCGANAnalysis of the DCGANGAN Training: Tips and TricksWasserstein GAN with Gradient Penalty (WGAN-GP)Wasserstein LossThe Lipschitz ConstraintEnforcing the Lipschitz ConstraintThe Gradient Penalty LossTraining the WGAN-GPAnalysis of the WGAN-GPConditional GAN (CGAN)CGAN ArchitectureTraining the CGANAnalysis of the CGANSummary
IntroductionLong Short-Term Memory Network (LSTM)The Recipes DatasetWorking with Text DataTokenizationCreating the Training SetThe LSTM ArchitectureThe Embedding LayerThe LSTM LayerThe LSTM CellTraining the LSTMAnalysis of the LSTMRecurrent Neural Network (RNN) ExtensionsStacked Recurrent NetworksGated Recurrent UnitsBidirectional CellsPixelCNNMasked Convolutional LayersResidual BlocksTraining the PixelCNNAnalysis of the PixelCNNMixture DistributionsSummary
IntroductionNormalizing FlowsChange of VariablesThe Jacobian DeterminantThe Change of Variables EquationRealNVPThe Two Moons DatasetCoupling LayersTraining the RealNVP ModelAnalysis of the RealNVP ModelOther Normalizing Flow ModelsGLOWFFJORDSummary

IntroductionEnergy-Based ModelsThe MNIST DatasetThe Energy FunctionSampling Using Langevin DynamicsTraining with Contrastive DivergenceAnalysis of the Energy-Based ModelOther Energy-Based ModelsSummary
IntroductionDenoising Diffusion Models (DDM)The Flowers DatasetThe Forward Diffusion ProcessThe Reparameterization TrickDiffusion SchedulesThe Reverse Diffusion ProcessThe U-Net Denoising ModelTraining the Diffusion ModelSampling from the Denoising Diffusion ModelAnalysis of the Diffusion ModelSummary
IntroductionGPTThe Wine Reviews DatasetAttentionQueries, Keys, and ValuesMultihead AttentionCausal MaskingThe Transformer BlockPositional EncodingTraining GPTAnalysis of GPTOther TransformersT5GPT-3 and GPT-4ChatGPTSummary
IntroductionProGANProgressive TrainingOutputsStyleGANThe Mapping NetworkThe Synthesis NetworkOutputs from StyleGANStyleGAN2Weight Modulation and DemodulationPath Length RegularizationNo Progressive GrowingOutputs from StyleGAN2Other Important GANsSelf-Attention GAN (SAGAN)BigGANVQ-GANViT VQ-GANSummary
IntroductionTransformers for Music GenerationThe Bach Cello Suite DatasetParsing MIDI FilesTokenizationCreating the Training SetSine Position EncodingMultiple Inputs and OutputsAnalysis of the Music-Generating TransformerTokenization of Polyphonic MusicMuseGANThe Bach Chorale DatasetThe MuseGAN GeneratorThe MuseGAN CriticAnalysis of the MuseGANSummary
IntroductionReinforcement LearningThe CarRacing EnvironmentWorld Model OverviewArchitectureTrainingCollecting Random Rollout DataTraining the VAEThe VAE ArchitectureExploring the VAECollecting Data to Train the MDN-RNNTraining the MDN-RNNThe MDN-RNN ArchitectureSampling from the MDN-RNNTraining the ControllerThe Controller ArchitectureCMA-ESParallelizing CMA-ESIn-Dream TrainingSummary
IntroductionDALL.E 2ArchitectureThe Text EncoderCLIPThe PriorThe DecoderExamples from DALL.E 2ImagenArchitectureDrawBenchExamples from ImagenStable DiffusionArchitectureExamples from Stable DiffusionFlamingoArchitectureThe Vision EncoderThe Perceiver ResamplerThe Language ModelExamples from FlamingoSummary
Timeline of Generative AI2014–2017: The VAE and GAN Era2018–2019: The Transformer Era2020–2022: The Big Model EraThe Current State of Generative AILarge Language ModelsText-to-Code ModelsText-to-Image ModelsOther ApplicationsThe Future of Generative AIGenerative AI in Everyday LifeGenerative AI in the WorkplaceGenerative AI in EducationGenerative AI Ethics and ChallengesFinal Thoughts

Content preview from Generative Deep Learning, 2nd Edition

Chapter 9. Transformers

Chapter Goals

In this chapter you will:

Learn about the origins of GPT, a powerful decoder Transformer model for text generation.
Learn conceptually how an attention mechanism mimics our way of attaching more importance to some words in a sentence than others.
Delve into how the attention mechanism works from first principles, including how queries, keys, and values are created and manipulated.
See the importance of causal masking for text generation tasks.
Understand how attention heads can be grouped into a multihead attention layer.
See how multihead attention layers form one part of a Transformer block that also includes layer normalization and skip connections.
Create positional encodings that capture the position of each token as well as the word token embedding.
Build a GPT model in Keras to generate the text contained in wine reviews.
Analyze the output from the GPT model, including interrogating the attention scores to inspect where the model is looking.
Learn about the different types of Transformers, including examples of the types of tasks that can be tackled by each and descriptions of the most famous state-of-the-art implementations.
Understand how encoder-decoder architectures work, like Google’s T5 model.
Explore the training process behind OpenAI’s ChatGPT.

We saw in Chapter 5 how we can build generative models on text data using recurrent neural networks (RNNs), such as LSTMs and GRUs. These autoregressive models ...