book

Grokking Deep Learning

by Andrew W. Trask

February 2019

Intermediate to advanced

336 pages

9h 29m

English

Manning Publications

Read now

Unlock full access

Who should read this bookRoadmapAbout the Code conventions and downloadsBook forum
Welcome to Grokking Deep LearningWhy you should learn deep learningWill this be difficult to learn?Why you should read this bookWhat you need to get startedYou’ll probably need some Python knowledgeSummary
What is deep learning?What is machine learning?Supervised machine learningUnsupervised machine learningParametric vs. nonparametric learningSupervised parametric learningUnsupervised parametric learningNonparametric learningSummary

Step 1: PredictA simple neural network making a predictionWhat is a neural network?What does this neural network do?Making a prediction with multiple inputsMultiple inputs: What does this neural network do?Multiple inputs: Complete runnable codeMaking a prediction with multiple outputsPredicting with multiple inputs and outputsMultiple inputs and outputs: How does it work?Predicting on predictionsA quick primer on NumPySummary
Predict, compare, and learnCompareLearnCompare: Does your network make good predictions?Why measure error?What’s the simplest form of neural learning?Hot and cold learningCharacteristics of hot and cold learningCalculating both direction and amount from errorOne iteration of gradient descentLearning is just reducing errorLet’s watch several steps of learningWhy does this work? What is weight_delta, really?Tunnel vision on one conceptA box with rods poking out of itDerivatives: Take twoWhat you really need to knowWhat you don’t really need to knowHow to use a derivative to learnLook familiar?Breaking gradient descentVisualizing the overcorrectionsDivergenceIntroducing alphaAlpha in codeMemorizing
Gradient descent learning with multiple inputsGradient descent with multiple inputs explainedLet’s watch several steps of learningFreezing one weight: What does it do?Gradient descent learning with multiple outputsGradient descent with multiple inputs and outputsWhat do these weights learn?Visualizing weight valuesVisualizing dot products (weighted sums)Summary
The streetlight problemPreparing the dataMatrices and the matrix relationshipCreating a matrix or two in PythonBuilding a neural networkLearning the whole datasetFull, batch, and stochastic gradient descentNeural networks learn correlationUp and down pressureEdge case: OverfittingEdge case: Conflicting pressureLearning indirect correlationCreating correlationStacking neural networks: A reviewBackpropagation: Long-distance error attributionBackpropagation: Why does this work?Linear vs. nonlinearWhy the neural network still doesn’t workThe secret to sometimes correlationA quick breakYour first deep neural networkBackpropagation in codeOne iteration of backpropagationPutting it all togetherWhy do deep networks matter?
It’s time to simplifyCorrelation summarizationThe previously overcomplicated visualizationThe simplified visualizationSimplifying even furtherLet’s see this network predictVisualizing using letters instead of picturesLinking the variablesEverything side by sideThe importance of visualization tools
Three-layer network on MNISTWell, that was easyMemorization vs. generalizationOverfitting in neural networksWhere overfitting comes fromThe simplest regularization: Early stoppingIndustry standard regularization: DropoutWhy dropout works: Ensembling worksDropout in codeDropout evaluated on MNISTBatch gradient descentSummary
What is an activation function?Standard hidden-layer activation functionsStandard output layer activation functionsThe core issue: Inputs have similaritysoftmax computationActivation installation instructionsMultiplying delta by the slopeConverting output to slope (derivative)Upgrading the MNIST network
Reusing weights in multiple placesThe convolutional layerA simple implementation in NumPySummary
What does it mean to understand language?Natural language processing (NLP)Supervised NLPIMDB movie reviews datasetCapturing word correlation in input dataPredicting movie reviewsIntro to an embedding layerInterpreting the outputNeural architectureComparing word embeddingsWhat is the meaning of a neuron?Filling in the blankMeaning is derived from lossKing – Man + Woman ~= QueenWord analogiesSummary
The challenge of arbitrary lengthDo comparisons really matter?The surprising power of averaged word vectorsHow is information stored in these embeddings?How does a neural network use embeddings?The limitations of bag-of-words vectorsUsing identity vectors to sum word embeddingsMatrices that change absolutely nothingLearning the transition matricesLearning to create useful sentence vectorsForward propagation in PythonHow do you backpropagate into this?Let’s train it!Setting things upForward propagation with arbitrary lengthBackpropagation with arbitrary lengthWeight update with arbitrary lengthExecution and output analysisSummary
What is a deep learning framework?Introduction to tensorsIntroduction to automatic gradient computation (autograd)A quick checkpointTensors that are used multiple timesUpgrading autograd to support multiuse tensorsHow does addition backpropagation work?Adding support for negationAdding support for additional functionsUsing autograd to train a neural networkAdding automatic optimizationAdding support for layer typesLayers that contain layersLoss-function layersHow to learn a frameworkNonlinearity layersThe embedding layerAdding indexing to autogradThe embedding layer (revisited)The cross-entropy layerThe recurrent neural network layerSummary
Character language modelingThe need for truncated backpropagationTruncated backpropagationA sample of the outputVanishing and exploding gradientsA toy example of RNN backpropagationLong short-term memory (LSTM) cellsSome intuition about LSTM gatesThe long short-term memory layerUpgrading the character language modelTraining the LSTM character language modelTuning the LSTM character language modelSummary
The problem of privacy in deep learningFederated learningLearning to detect spamLet’s make it federatedHacking into federated learningSecure aggregationHomomorphic encryptionHomomorphically encrypted federated learningSummary
Congratulations!Step 1: Start learning PyTorchStep 2: Start another deep learning courseStep 3: Grab a mathy deep learning textbookStep 4: Start a blog, and teach deep learningStep 5: TwitterStep 6: Implement academic papersStep 7: Acquire access to a GPU (or many)Step 8: Get paid to practiceStep 9: Join an open source projectStep 10: Develop your local community

Content preview from Grokking Deep Learning

Chapter 14. Learning to write like Shakespeare: long short-term memory

In this chapter

Character language modeling
Truncated backpropagation
Vanishing and exploding gradients
A toy example of RNN backpropagation
Long short-term memory (LSTM) cells

“Lord, what fools these mortals be!”

William Shakespeare A Midsummer Night’s Dream

Character language modeling

Let’s tackle a more challenging task with the RNN

At the end of chapters 12 and 13, you trained vanilla recurrent neural networks (RNNs) that learned a simple series prediction problem. But you were training over a toy dataset of phrases that were synthetically generated using rules.

In this chapter, you’ll attempt language modeling over a much more challenging dataset: the works of ...