book

Generative Deep Learning

by David Foster

June 2019

Intermediate to advanced

327 pages

7h 36m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Objective and ApproachPrerequisitesOther ResourcesConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
What Is Generative Modeling?Generative Versus Discriminative ModelingAdvances in Machine LearningThe Rise of Generative ModelingThe Generative Modeling FrameworkProbabilistic Generative ModelsHello Wrodl!Your First Probabilistic Generative ModelNaive BayesHello Wrodl! ContinuedThe Challenges of Generative ModelingRepresentation LearningSetting Up Your EnvironmentSummary
Structured and Unstructured DataDeep Neural NetworksKeras and TensorFlowYour First Deep Neural NetworkLoading the DataBuilding the ModelCompiling the ModelTraining the ModelEvaluating the ModelImproving the ModelConvolutional LayersBatch NormalizationDropout LayersPutting It All TogetherSummary
The Art ExhibitionAutoencodersYour First AutoencoderThe EncoderThe DecoderJoining the Encoder to the DecoderAnalysis of the AutoencoderThe Variational Art ExhibitionBuilding a Variational AutoencoderThe EncoderThe Loss FunctionAnalysis of the Variational AutoencoderUsing VAEs to Generate FacesTraining the VAEAnalysis of the VAEGenerating New FacesLatent Space ArithmeticMorphing Between FacesSummary
GanimalsIntroduction to GANsYour First GANThe DiscriminatorThe GeneratorTraining the GANGAN ChallengesOscillating LossMode CollapseUninformative LossHyperparametersTackling the GAN ChallengesWasserstein GANWasserstein LossThe Lipschitz ConstraintWeight ClippingTraining the WGANAnalysis of the WGANWGAN-GPThe Gradient Penalty LossAnalysis of WGAN-GPSummary
Apples and OrgangesCycleGANYour First CycleGANOverviewThe Generators (U-Net)The DiscriminatorsCompiling the CycleGANTraining the CycleGANAnalysis of the CycleGANCreating a CycleGAN to Paint Like MonetThe Generators (ResNet)Analysis of the CycleGANNeural Style TransferContent LossStyle LossTotal Variance LossRunning the Neural Style TransferAnalysis of the Neural Style Transfer ModelSummary
The Literary Society for Troublesome MiscreantsLong Short-Term Memory NetworksYour First LSTM NetworkTokenizationBuilding the DatasetThe LSTM ArchitectureThe Embedding LayerThe LSTM LayerThe LSTM CellGenerating New TextRNN ExtensionsStacked Recurrent NetworksGated Recurrent UnitsBidirectional CellsEncoder–Decoder ModelsA Question and Answer GeneratorA Question-Answer DatasetModel ArchitectureInferenceModel ResultsSummary
PreliminariesMusical NotationYour First Music-Generating RNNAttentionBuilding an Attention Mechanism in KerasAnalysis of the RNN with AttentionAttention in Encoder–Decoder NetworksGenerating Polyphonic MusicThe Musical OrganYour First MuseGANThe MuseGAN GeneratorChords, Style, Melody, and GrooveThe Bar GeneratorPutting It All TogetherThe CriticAnalysis of the MuseGANSummary

Reinforcement LearningOpenAI GymWorld Model ArchitectureThe Variational AutoencoderThe MDN-RNNThe ControllerSetupTraining Process OverviewCollecting Random Rollout DataTraining the VAEThe VAE ArchitectureExploring the VAECollecting Data to Train the RNNTraining the MDN-RNNThe MDN-RNN ArchitectureSampling the Next z and Reward from the MDN-RNNThe MDN-RNN Loss FunctionTraining the ControllerThe Controller ArchitectureCMA-ESParallelizing CMA-ESOutput from the Controller TrainingIn-Dream TrainingIn-Dream Training the ControllerChallenges of In-Dream TrainingSummary
Five Years of ProgressThe TransformerPositional EncodingMultihead AttentionThe DecoderAnalysis of the TransformerBERTGPT-2MuseNetAdvances in Image GenerationProGANSelf-Attention GAN (SAGAN)BigGANStyleGANApplications of Generative ModelingAI ArtAI Music

Content preview from Generative Deep Learning

Chapter 3. Variational Autoencoders

In 2013, Diederik P. Kingma and Max Welling published a paper that laid the foundations for a type of neural network known as a variational autoencoder (VAE).¹ This is now one of the most fundamental and well-known deep learning architectures for generative modeling. In this chapter, we shall start by building a standard autoencoder and then see how we can extend this framework to develop a variational autoencoder—our first example of a generative deep learning model.

Along the way, we will pick apart both types of model, to understand how they work at a granular level. By the end of the chapter you should have a complete understanding of how to build and manipulate autoencoder-based models and, in particular, how to build a variational autoencoder from scratch to generate images based on your own training set.

Let’s start by paying a visit to a strange art exhibition…

The Art Exhibition

Two brothers, Mr. N. Coder and Mr. D. Coder, run an art gallery. One weekend, they host an exhibition focused on monochrome studies of single-digit numbers. The exhibition is particularly strange because it contains only one wall and no physical artwork. When a new painting arrives for display, Mr. N. Coder simply chooses a point on the wall to represent the painting, places a marker at this point, then throws the original artwork away. When a customer requests to see the painting, Mr. D. Coder attempts to re-create the artwork using just the coordinates of ...