book

Fundamentals of Deep Learning, 2nd Edition

by Nithin Buduma, Nikhil Buduma, Joe Papa

May 2022

Intermediate to advanced

387 pages

11h 47m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Prerequisites and ObjectivesHow Is This Book Organized?Conventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgementsNithin and NikhilJoe
Data Structures and OperationsMatrix OperationsVector OperationsMatrix-Vector MultiplicationThe Fundamental SpacesThe Column SpaceThe Null SpaceEigenvectors and EigenvaluesSummary
Events and ProbabilityConditional ProbabilityRandom VariablesExpectationVarianceBayes’ TheoremEntropy, Cross Entropy, and KL DivergenceContinuous Probability DistributionsSummary
Building Intelligent MachinesThe Limits of Traditional Computer ProgramsThe Mechanics of Machine LearningThe NeuronExpressing Linear Perceptrons as NeuronsFeed-Forward Neural NetworksLinear Neurons and Their LimitationsSigmoid, Tanh, and ReLU NeuronsSoftmax Output LayersSummary
The Fast-Food ProblemGradient DescentThe Delta Rule and Learning RatesGradient Descent with Sigmoidal NeuronsThe Backpropagation AlgorithmStochastic and Minibatch Gradient DescentTest Sets, Validation Sets, and OverfittingPreventing Overfitting in Deep Neural NetworksSummary
Introduction to PyTorchInstalling PyTorchPyTorch TensorsTensor InitTensor AttributesTensor OperationsGradients in PyTorchThe PyTorch nn ModulePyTorch Datasets and DataloadersBuilding the MNIST Classifier in PyTorchSummary
The Challenges with Gradient DescentLocal Minima in the Error Surfaces of Deep NetworksModel IdentifiabilityHow Pesky Are Spurious Local Minima in Deep Networks?Flat Regions in the Error SurfaceWhen the Gradient Points in the Wrong DirectionMomentum-Based OptimizationA Brief View of Second-Order MethodsLearning Rate AdaptationAdaGrad—Accumulating Historical GradientsRMSProp—Exponentially Weighted Moving Average of GradientsAdam—Combining Momentum and RMSPropThe Philosophy Behind Optimizer SelectionSummary
Neurons in Human VisionThe Shortcomings of Feature SelectionVanilla Deep Neural Networks Don’t ScaleFilters and Feature MapsFull Description of the Convolutional LayerMax PoolingFull Architectural Description of Convolution NetworksClosing the Loop on MNIST with Convolutional NetworksImage Preprocessing Pipelines Enable More Robust ModelsAccelerating Training with Batch NormalizationGroup Normalization for Memory Constrained Learning TasksBuilding a Convolutional Network for CIFAR-10Visualizing Learning in Convolutional NetworksResidual Learning and Skip Connections for Very Deep NetworksBuilding a Residual Network with Superhuman VisionLeveraging Convolutional Filters to Replicate Artistic StylesLearning Convolutional Filters for Other Problem DomainsSummary
Learning Lower-Dimensional RepresentationsPrincipal Component AnalysisMotivating the Autoencoder ArchitectureImplementing an Autoencoder in PyTorchDenoising to Force Robust RepresentationsSparsity in AutoencodersWhen Context Is More Informative than the Input VectorThe Word2Vec FrameworkImplementing the Skip-Gram ArchitectureSummary
Analyzing Variable-Length InputsTackling seq2seq with Neural N-GramsImplementing a Part-of-Speech TaggerDependency Parsing and SyntaxNetBeam Search and Global NormalizationA Case for Stateful Deep Learning ModelsRecurrent Neural NetworksThe Challenges with Vanishing GradientsLong Short-Term Memory UnitsPyTorch Primitives for RNN ModelsImplementing a Sentiment Analysis ModelSolving seq2seq Tasks with Recurrent Neural NetworksAugmenting Recurrent Networks with AttentionDissecting a Neural Translation NetworkSelf-Attention and TransformersSummary

Generative Adversarial NetworksVariational AutoencodersImplementing a VAEScore-Based Generative ModelsDenoising Autoencoders and Score MatchingSummary
OverviewDecision Trees and Tree-Based AlgorithmsLinear RegressionMethods for Evaluating Feature ImportancePermutation Feature ImportancePartial Dependence PlotsExtractive RationalizationLIMESHAPSummary
Neural Turing MachinesAttention-Based Memory AccessNTM Memory Addressing MechanismsDifferentiable Neural ComputersInterference-Free Writing in DNCsDNC Memory ReuseTemporal Linking of DNC WritesUnderstanding the DNC Read HeadThe DNC Controller NetworkVisualizing the DNC in ActionImplementing the DNC in PyTorchTeaching a DNC to Read and ComprehendSummary
Deep Reinforcement Learning Masters Atari GamesWhat Is Reinforcement Learning?Markov Decision ProcessesPolicyFuture ReturnDiscounted Future ReturnExplore Versus Exploit ϵ -GreedyAnnealed ϵ -GreedyPolicy Versus Value LearningPole-Cart with Policy GradientsOpenAI GymCreating an AgentBuilding the Model and OptimizerSampling ActionsKeeping Track of HistoryPolicy Gradient Main FunctionPGAgent Performance on Pole-CartTrust-Region Policy OptimizationProximal Policy OptimizationQ-Learning and Deep Q-NetworksThe Bellman EquationIssues with Value IterationApproximating the Q-FunctionDeep Q-NetworkTraining DQNLearning StabilityTarget Q-NetworkExperience ReplayFrom Q-Function to PolicyDQN and the Markov AssumptionDQN’s Solution to the Markov AssumptionPlaying Breakout with DQNBuilding Our ArchitectureStacking FramesSetting Up Training OperationsUpdating Our Target Q-NetworkImplementing Experience ReplayDQN Main LoopDQNAgent Results on BreakoutImproving and Moving Beyond DQNDeep Recurrent Q-NetworksAsynchronous Advantage Actor-Critic AgentUNsupervised REinforcement and Auxiliary LearningSummary

Content preview from Fundamentals of Deep Learning, 2nd Edition

Chapter 9. Models for Sequence Analysis

Surya Bhupatiraju

Analyzing Variable-Length Inputs

Up until now, we’ve worked only with data with fixed sizes: images from MNIST, CIFAR-10, and ImageNet. These models are incredibly powerful, but there are many situations in which fixed-length models are insufficient. The vast majority of interactions in our daily lives require a deep understanding of sequences—whether it’s reading the morning newspaper, making a bowl of cereal, listening to the radio, watching a presentation, or deciding to execute a trade on the stock market. To adapt to variable-length inputs, we’ll have to be a little bit more clever about how we approach designing deep learning models.

Figure 9-1 illustrates how our feed-forward neural networks break when analyzing sequences. If the sequence is the same size as the input layer, the model can perform as expected. It’s even possible to deal with smaller inputs by padding zeros to the end of the input until it’s the appropriate length. However, the moment the input exceeds the size of the input layer, naively using the feed-forward network no longer works.

Feed-forward networks thrive on fixed input size problems. Zero padding can address the handling of smaller inputs, but when naively utilized, these models break when inputs exceed the fixed input size.