book

Fundamentals of Deep Learning, 2nd Edition

by Nithin Buduma, Nikhil Buduma, Joe Papa

May 2022

Intermediate to advanced

387 pages

11h 47m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Prerequisites and ObjectivesHow Is This Book Organized?Conventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgementsNithin and NikhilJoe
Data Structures and OperationsMatrix OperationsVector OperationsMatrix-Vector MultiplicationThe Fundamental SpacesThe Column SpaceThe Null SpaceEigenvectors and EigenvaluesSummary
Events and ProbabilityConditional ProbabilityRandom VariablesExpectationVarianceBayes’ TheoremEntropy, Cross Entropy, and KL DivergenceContinuous Probability DistributionsSummary
Building Intelligent MachinesThe Limits of Traditional Computer ProgramsThe Mechanics of Machine LearningThe NeuronExpressing Linear Perceptrons as NeuronsFeed-Forward Neural NetworksLinear Neurons and Their LimitationsSigmoid, Tanh, and ReLU NeuronsSoftmax Output LayersSummary
The Fast-Food ProblemGradient DescentThe Delta Rule and Learning RatesGradient Descent with Sigmoidal NeuronsThe Backpropagation AlgorithmStochastic and Minibatch Gradient DescentTest Sets, Validation Sets, and OverfittingPreventing Overfitting in Deep Neural NetworksSummary
Introduction to PyTorchInstalling PyTorchPyTorch TensorsTensor InitTensor AttributesTensor OperationsGradients in PyTorchThe PyTorch nn ModulePyTorch Datasets and DataloadersBuilding the MNIST Classifier in PyTorchSummary
The Challenges with Gradient DescentLocal Minima in the Error Surfaces of Deep NetworksModel IdentifiabilityHow Pesky Are Spurious Local Minima in Deep Networks?Flat Regions in the Error SurfaceWhen the Gradient Points in the Wrong DirectionMomentum-Based OptimizationA Brief View of Second-Order MethodsLearning Rate AdaptationAdaGrad—Accumulating Historical GradientsRMSProp—Exponentially Weighted Moving Average of GradientsAdam—Combining Momentum and RMSPropThe Philosophy Behind Optimizer SelectionSummary
Neurons in Human VisionThe Shortcomings of Feature SelectionVanilla Deep Neural Networks Don’t ScaleFilters and Feature MapsFull Description of the Convolutional LayerMax PoolingFull Architectural Description of Convolution NetworksClosing the Loop on MNIST with Convolutional NetworksImage Preprocessing Pipelines Enable More Robust ModelsAccelerating Training with Batch NormalizationGroup Normalization for Memory Constrained Learning TasksBuilding a Convolutional Network for CIFAR-10Visualizing Learning in Convolutional NetworksResidual Learning and Skip Connections for Very Deep NetworksBuilding a Residual Network with Superhuman VisionLeveraging Convolutional Filters to Replicate Artistic StylesLearning Convolutional Filters for Other Problem DomainsSummary
Learning Lower-Dimensional RepresentationsPrincipal Component AnalysisMotivating the Autoencoder ArchitectureImplementing an Autoencoder in PyTorchDenoising to Force Robust RepresentationsSparsity in AutoencodersWhen Context Is More Informative than the Input VectorThe Word2Vec FrameworkImplementing the Skip-Gram ArchitectureSummary
Analyzing Variable-Length InputsTackling seq2seq with Neural N-GramsImplementing a Part-of-Speech TaggerDependency Parsing and SyntaxNetBeam Search and Global NormalizationA Case for Stateful Deep Learning ModelsRecurrent Neural NetworksThe Challenges with Vanishing GradientsLong Short-Term Memory UnitsPyTorch Primitives for RNN ModelsImplementing a Sentiment Analysis ModelSolving seq2seq Tasks with Recurrent Neural NetworksAugmenting Recurrent Networks with AttentionDissecting a Neural Translation NetworkSelf-Attention and TransformersSummary

Generative Adversarial NetworksVariational AutoencodersImplementing a VAEScore-Based Generative ModelsDenoising Autoencoders and Score MatchingSummary
OverviewDecision Trees and Tree-Based AlgorithmsLinear RegressionMethods for Evaluating Feature ImportancePermutation Feature ImportancePartial Dependence PlotsExtractive RationalizationLIMESHAPSummary
Neural Turing MachinesAttention-Based Memory AccessNTM Memory Addressing MechanismsDifferentiable Neural ComputersInterference-Free Writing in DNCsDNC Memory ReuseTemporal Linking of DNC WritesUnderstanding the DNC Read HeadThe DNC Controller NetworkVisualizing the DNC in ActionImplementing the DNC in PyTorchTeaching a DNC to Read and ComprehendSummary
Deep Reinforcement Learning Masters Atari GamesWhat Is Reinforcement Learning?Markov Decision ProcessesPolicyFuture ReturnDiscounted Future ReturnExplore Versus Exploit ϵ -GreedyAnnealed ϵ -GreedyPolicy Versus Value LearningPole-Cart with Policy GradientsOpenAI GymCreating an AgentBuilding the Model and OptimizerSampling ActionsKeeping Track of HistoryPolicy Gradient Main FunctionPGAgent Performance on Pole-CartTrust-Region Policy OptimizationProximal Policy OptimizationQ-Learning and Deep Q-NetworksThe Bellman EquationIssues with Value IterationApproximating the Q-FunctionDeep Q-NetworkTraining DQNLearning StabilityTarget Q-NetworkExperience ReplayFrom Q-Function to PolicyDQN and the Markov AssumptionDQN’s Solution to the Markov AssumptionPlaying Breakout with DQNBuilding Our ArchitectureStacking FramesSetting Up Training OperationsUpdating Our Target Q-NetworkImplementing Experience ReplayDQN Main LoopDQNAgent Results on BreakoutImproving and Moving Beyond DQNDeep Recurrent Q-NetworksAsynchronous Advantage Actor-Critic AgentUNsupervised REinforcement and Auxiliary LearningSummary

Content preview from Fundamentals of Deep Learning, 2nd Edition

Chapter 6. Beyond Gradient Descent

The Challenges with Gradient Descent

The fundamental ideas behind neural networks have existed for decades, but it wasn’t until recently that neural network-based learning models have become mainstream. Our fascination with neural networks has everything to do with their expressiveness, a quality we’ve unlocked by creating networks with many layers. As we have discussed in previous chapters, deep neural networks are able to crack problems that were previously deemed intractable. Training deep neural networks end to end, however, is fraught with difficult challenges that took many technological innovations to unravel, including massive labeled datasets (ImageNet, CIFAR-10, etc.), better hardware in the form of GPU acceleration, and several algorithmic discoveries.

For several years, researchers resorted to layer-wise greedy pretraining to grapple with the complex error surfaces presented by deep learning models.¹ These time-intensive strategies would try to find more accurate initializations for the model’s parameters one layer at a time before using minibatch gradient descent to converge to the optimal parameter settings. More recently, however, breakthroughs in optimization methods have enabled us to train models directly in an end-to-end fashion.

In this chapter, we will discuss several of these breakthroughs. The next couple of sections will focus primarily on local minima and whether they pose hurdles for successfully training deep models. ...