book

Generative Deep Learning, 2nd Edition

by David Foster

April 2023

Intermediate to advanced

456 pages

11h 12m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Includes

Includes Quizzes

Objective and ApproachPrerequisitesRoadmapChanges in the Second EditionOther ResourcesConventions Used in This BookCodebaseUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
What Is Generative Modeling?Generative Versus Discriminative ModelingThe Rise of Generative ModelingGenerative Modeling and AIOur First Generative ModelHello World!The Generative Modeling FrameworkRepresentation LearningCore Probability TheoryGenerative Model TaxonomyThe Generative Deep Learning CodebaseCloning the RepositoryUsing DockerRunning on a GPUSummary
Data for Deep LearningDeep Neural NetworksWhat Is a Neural Network?Learning High-Level FeaturesTensorFlow and KerasMultilayer Perceptron (MLP)Preparing the DataBuilding the ModelCompiling the ModelTraining the ModelEvaluating the ModelConvolutional Neural Network (CNN)Convolutional LayersBatch NormalizationDropoutBuilding the CNNTraining and Evaluating the CNNSummary
IntroductionAutoencodersThe Fashion-MNIST DatasetThe Autoencoder ArchitectureThe EncoderThe DecoderJoining the Encoder to the DecoderReconstructing ImagesVisualizing the Latent SpaceGenerating New ImagesVariational AutoencodersThe EncoderThe Loss FunctionTraining the Variational AutoencoderAnalysis of the Variational AutoencoderExploring the Latent SpaceThe CelebA DatasetTraining the Variational AutoencoderAnalysis of the Variational AutoencoderGenerating New FacesLatent Space ArithmeticMorphing Between FacesSummary
IntroductionDeep Convolutional GAN (DCGAN)The Bricks DatasetThe DiscriminatorThe GeneratorTraining the DCGANAnalysis of the DCGANGAN Training: Tips and TricksWasserstein GAN with Gradient Penalty (WGAN-GP)Wasserstein LossThe Lipschitz ConstraintEnforcing the Lipschitz ConstraintThe Gradient Penalty LossTraining the WGAN-GPAnalysis of the WGAN-GPConditional GAN (CGAN)CGAN ArchitectureTraining the CGANAnalysis of the CGANSummary
IntroductionLong Short-Term Memory Network (LSTM)The Recipes DatasetWorking with Text DataTokenizationCreating the Training SetThe LSTM ArchitectureThe Embedding LayerThe LSTM LayerThe LSTM CellTraining the LSTMAnalysis of the LSTMRecurrent Neural Network (RNN) ExtensionsStacked Recurrent NetworksGated Recurrent UnitsBidirectional CellsPixelCNNMasked Convolutional LayersResidual BlocksTraining the PixelCNNAnalysis of the PixelCNNMixture DistributionsSummary
IntroductionNormalizing FlowsChange of VariablesThe Jacobian DeterminantThe Change of Variables EquationRealNVPThe Two Moons DatasetCoupling LayersTraining the RealNVP ModelAnalysis of the RealNVP ModelOther Normalizing Flow ModelsGLOWFFJORDSummary

IntroductionEnergy-Based ModelsThe MNIST DatasetThe Energy FunctionSampling Using Langevin DynamicsTraining with Contrastive DivergenceAnalysis of the Energy-Based ModelOther Energy-Based ModelsSummary
IntroductionDenoising Diffusion Models (DDM)The Flowers DatasetThe Forward Diffusion ProcessThe Reparameterization TrickDiffusion SchedulesThe Reverse Diffusion ProcessThe U-Net Denoising ModelTraining the Diffusion ModelSampling from the Denoising Diffusion ModelAnalysis of the Diffusion ModelSummary
IntroductionGPTThe Wine Reviews DatasetAttentionQueries, Keys, and ValuesMultihead AttentionCausal MaskingThe Transformer BlockPositional EncodingTraining GPTAnalysis of GPTOther TransformersT5GPT-3 and GPT-4ChatGPTSummary
IntroductionProGANProgressive TrainingOutputsStyleGANThe Mapping NetworkThe Synthesis NetworkOutputs from StyleGANStyleGAN2Weight Modulation and DemodulationPath Length RegularizationNo Progressive GrowingOutputs from StyleGAN2Other Important GANsSelf-Attention GAN (SAGAN)BigGANVQ-GANViT VQ-GANSummary
IntroductionTransformers for Music GenerationThe Bach Cello Suite DatasetParsing MIDI FilesTokenizationCreating the Training SetSine Position EncodingMultiple Inputs and OutputsAnalysis of the Music-Generating TransformerTokenization of Polyphonic MusicMuseGANThe Bach Chorale DatasetThe MuseGAN GeneratorThe MuseGAN CriticAnalysis of the MuseGANSummary
IntroductionReinforcement LearningThe CarRacing EnvironmentWorld Model OverviewArchitectureTrainingCollecting Random Rollout DataTraining the VAEThe VAE ArchitectureExploring the VAECollecting Data to Train the MDN-RNNTraining the MDN-RNNThe MDN-RNN ArchitectureSampling from the MDN-RNNTraining the ControllerThe Controller ArchitectureCMA-ESParallelizing CMA-ESIn-Dream TrainingSummary
IntroductionDALL.E 2ArchitectureThe Text EncoderCLIPThe PriorThe DecoderExamples from DALL.E 2ImagenArchitectureDrawBenchExamples from ImagenStable DiffusionArchitectureExamples from Stable DiffusionFlamingoArchitectureThe Vision EncoderThe Perceiver ResamplerThe Language ModelExamples from FlamingoSummary
Timeline of Generative AI2014–2017: The VAE and GAN Era2018–2019: The Transformer Era2020–2022: The Big Model EraThe Current State of Generative AILarge Language ModelsText-to-Code ModelsText-to-Image ModelsOther ApplicationsThe Future of Generative AIGenerative AI in Everyday LifeGenerative AI in the WorkplaceGenerative AI in EducationGenerative AI Ethics and ChallengesFinal Thoughts

Content preview from Generative Deep Learning, 2nd Edition

Chapter 10. Advanced GANs

Chapter Goals

In this chapter you will:

See how a ProGAN model progressively trains a GAN to generate high-resolution images.
Understand how ProGAN was adapted to build StyleGAN, a high-performing GAN for image synthesis.
Explore how StyleGAN was adjusted to create StyleGAN2, a state-of-the-art model that improves further upon the original work.
Learn about the key contributions of these models, including progressive training, adaptive instance normalization, weight modulation and demodulation, and path length regularization.
Walk through the architecture of the Self-Attention GAN (SAGAN), which incorporates the attention mechanism into the GAN framework.
See how BigGAN expands upon the ideas in the SAGAN paper to produce high-quality images.
Learn how VQ-GAN uses a codebook to encode images into a discrete sequence of tokens that can be modeled using a Transformer.
See how ViT VQ-GAN adapts the VQ-GAN architecture to use Transformers instead of convolutional layers in the encoder and decoder.

Chapter 4 introduced generative adversarial networks (GANs), a class of generative model that has produced state-of-the-art results across a wide variety of image generation tasks. The flexibility in the model architecture and training process has led academics and deep learning practitioners to find new ways to design and train GANs, leading to many different advanced flavors of the architecture that we shall explore in this chapter.