book

Grokking Deep Learning

Name: Grokking Deep Learning
Author: Andrew W. Trask
ISBN: 9781617293702

by Andrew W. Trask

February 2019

Intermediate to advanced

336 pages

9h 29m

English

Manning Publications

Audiobook available

Read now

Unlock full access

Copyright
Dedication
Brief Table of Contents
Table of Contents
Preface
Acknowledgments
About this book
Who should read this bookRoadmapAbout the Code conventions and downloadsBook forum
About the author
Chapter 1. Introducing deep learning: why you should learn it
Welcome to Grokking Deep LearningWhy you should learn deep learningWill this be difficult to learn?Why you should read this bookWhat you need to get startedYou’ll probably need some Python knowledgeSummary
Chapter 2. Fundamental concepts: how do machines learn?
What is deep learning?What is machine learning?Supervised machine learningUnsupervised machine learningParametric vs. nonparametric learningSupervised parametric learningUnsupervised parametric learningNonparametric learningSummary

Chapter 3. Introduction to neural prediction: forward propagation
Step 1: PredictA simple neural network making a predictionWhat is a neural network?What does this neural network do?Making a prediction with multiple inputsMultiple inputs: What does this neural network do?Multiple inputs: Complete runnable codeMaking a prediction with multiple outputsPredicting with multiple inputs and outputsMultiple inputs and outputs: How does it work?Predicting on predictionsA quick primer on NumPySummary
Chapter 4. Introduction to neural learning: gradient descent
Predict, compare, and learnCompareLearnCompare: Does your network make good predictions?Why measure error?What’s the simplest form of neural learning?Hot and cold learningCharacteristics of hot and cold learningCalculating both direction and amount from errorOne iteration of gradient descentLearning is just reducing errorLet’s watch several steps of learningWhy does this work? What is weight_delta, really?Tunnel vision on one conceptA box with rods poking out of itDerivatives: Take twoWhat you really need to knowWhat you don’t really need to knowHow to use a derivative to learnLook familiar?Breaking gradient descentVisualizing the overcorrectionsDivergenceIntroducing alphaAlpha in codeMemorizing
Chapter 5. Learning multiple weights at a time: generalizing gradient descent
Gradient descent learning with multiple inputsGradient descent with multiple inputs explainedLet’s watch several steps of learningFreezing one weight: What does it do?Gradient descent learning with multiple outputsGradient descent with multiple inputs and outputsWhat do these weights learn?Visualizing weight valuesVisualizing dot products (weighted sums)Summary
Chapter 6. Building your first deep neural network: introduction to backpropagation
The streetlight problemPreparing the dataMatrices and the matrix relationshipCreating a matrix or two in PythonBuilding a neural networkLearning the whole datasetFull, batch, and stochastic gradient descentNeural networks learn correlationUp and down pressureEdge case: OverfittingEdge case: Conflicting pressureLearning indirect correlationCreating correlationStacking neural networks: A reviewBackpropagation: Long-distance error attributionBackpropagation: Why does this work?Linear vs. nonlinearWhy the neural network still doesn’t workThe secret to sometimes correlationA quick breakYour first deep neural networkBackpropagation in codeOne iteration of backpropagationPutting it all togetherWhy do deep networks matter?
Chapter 7. How to picture neural networks: in your head and on paper
It’s time to simplifyCorrelation summarizationThe previously overcomplicated visualizationThe simplified visualizationSimplifying even furtherLet’s see this network predictVisualizing using letters instead of picturesLinking the variablesEverything side by sideThe importance of visualization tools
Chapter 8. Learning signal and ignoring noise: introduction to regularization and batching
Three-layer network on MNISTWell, that was easyMemorization vs. generalizationOverfitting in neural networksWhere overfitting comes fromThe simplest regularization: Early stoppingIndustry standard regularization: DropoutWhy dropout works: Ensembling worksDropout in codeDropout evaluated on MNISTBatch gradient descentSummary
Chapter 9. Modeling probabilities and nonlinearities: activation functions
What is an activation function?Standard hidden-layer activation functionsStandard output layer activation functionsThe core issue: Inputs have similaritysoftmax computationActivation installation instructionsMultiplying delta by the slopeConverting output to slope (derivative)Upgrading the MNIST network
Chapter 10. Neural learning about edges and corners: intro to convolutional neural networks
Reusing weights in multiple placesThe convolutional layerA simple implementation in NumPySummary
Chapter 11. Neural networks that understand language: king – man + woman == ?
What does it mean to understand language?Natural language processing (NLP)Supervised NLPIMDB movie reviews datasetCapturing word correlation in input dataPredicting movie reviewsIntro to an embedding layerInterpreting the outputNeural architectureComparing word embeddingsWhat is the meaning of a neuron?Filling in the blankMeaning is derived from lossKing – Man + Woman ~= QueenWord analogiesSummary
Chapter 12. Neural networks that write like Shakespeare: recurrent layers for variable-length data
The challenge of arbitrary lengthDo comparisons really matter?The surprising power of averaged word vectorsHow is information stored in these embeddings?How does a neural network use embeddings?The limitations of bag-of-words vectorsUsing identity vectors to sum word embeddingsMatrices that change absolutely nothingLearning the transition matricesLearning to create useful sentence vectorsForward propagation in PythonHow do you backpropagate into this?Let’s train it!Setting things upForward propagation with arbitrary lengthBackpropagation with arbitrary lengthWeight update with arbitrary lengthExecution and output analysisSummary
Chapter 13. Introducing automatic optimization: let’s build a deep learning framework
What is a deep learning framework?Introduction to tensorsIntroduction to automatic gradient computation (autograd)A quick checkpointTensors that are used multiple timesUpgrading autograd to support multiuse tensorsHow does addition backpropagation work?Adding support for negationAdding support for additional functionsUsing autograd to train a neural networkAdding automatic optimizationAdding support for layer typesLayers that contain layersLoss-function layersHow to learn a frameworkNonlinearity layersThe embedding layerAdding indexing to autogradThe embedding layer (revisited)The cross-entropy layerThe recurrent neural network layerSummary
Chapter 14. Learning to write like Shakespeare: long short-term memory
Character language modelingThe need for truncated backpropagationTruncated backpropagationA sample of the outputVanishing and exploding gradientsA toy example of RNN backpropagationLong short-term memory (LSTM) cellsSome intuition about LSTM gatesThe long short-term memory layerUpgrading the character language modelTraining the LSTM character language modelTuning the LSTM character language modelSummary
Chapter 15. Deep learning on unseen data: introducing federated learning
The problem of privacy in deep learningFederated learningLearning to detect spamLet’s make it federatedHacking into federated learningSecure aggregationHomomorphic encryptionHomomorphically encrypted federated learningSummary
Chapter 16. Where to go from here: a brief guide
Congratulations!Step 1: Start learning PyTorchStep 2: Start another deep learning courseStep 3: Grab a mathy deep learning textbookStep 4: Start a blog, and teach deep learningStep 5: TwitterStep 6: Implement academic papersStep 7: Acquire access to a GPU (or many)Step 8: Get paid to practiceStep 9: Join an open source projectStep 10: Develop your local community
Index

Content preview from Grokking Deep Learning

Chapter 9. Modeling probabilities and nonlinearities: activation functions

In this chapter

What is an activation function?

Standard hidden activation functions

Sigmoid
Tanh

Standard output activation functions

Softmax

Activation function installation instructions

“I know that 2 and 2 make 4—& should be glad to prove it too if I could—though I must say if by any sort of process I could convert 2 & 2 into five it would give me much greater pleasure.”

George Gordon Byron, letter to Annabella Milbanke, November 10, 1813

What is an activation function?

It’s a function applied to the neurons in a layer during prediction

An activation function is a function applied to the neurons in a layer during prediction. This should seem very familiar, ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617293702Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Grokking Deep Learning

by Andrew W. Trask