book

Grokking Deep Learning

Name: Grokking Deep Learning
Author: Andrew W. Trask
ISBN: 9781617293702

by Andrew W. Trask

February 2019

Intermediate to advanced

336 pages

9h 29m

English

Manning Publications

Read now

Unlock full access

Copyright
Dedication
Brief Table of Contents
Table of Contents
Preface
Acknowledgments
About this book
Who should read this bookRoadmapAbout the Code conventions and downloadsBook forum
About the author
Chapter 1. Introducing deep learning: why you should learn it
Welcome to Grokking Deep LearningWhy you should learn deep learningWill this be difficult to learn?Why you should read this bookWhat you need to get startedYou’ll probably need some Python knowledgeSummary
Chapter 2. Fundamental concepts: how do machines learn?
What is deep learning?What is machine learning?Supervised machine learningUnsupervised machine learningParametric vs. nonparametric learningSupervised parametric learningUnsupervised parametric learningNonparametric learningSummary

Chapter 3. Introduction to neural prediction: forward propagation
Step 1: PredictA simple neural network making a predictionWhat is a neural network?What does this neural network do?Making a prediction with multiple inputsMultiple inputs: What does this neural network do?Multiple inputs: Complete runnable codeMaking a prediction with multiple outputsPredicting with multiple inputs and outputsMultiple inputs and outputs: How does it work?Predicting on predictionsA quick primer on NumPySummary
Chapter 4. Introduction to neural learning: gradient descent
Predict, compare, and learnCompareLearnCompare: Does your network make good predictions?Why measure error?What’s the simplest form of neural learning?Hot and cold learningCharacteristics of hot and cold learningCalculating both direction and amount from errorOne iteration of gradient descentLearning is just reducing errorLet’s watch several steps of learningWhy does this work? What is weight_delta, really?Tunnel vision on one conceptA box with rods poking out of itDerivatives: Take twoWhat you really need to knowWhat you don’t really need to knowHow to use a derivative to learnLook familiar?Breaking gradient descentVisualizing the overcorrectionsDivergenceIntroducing alphaAlpha in codeMemorizing
Chapter 5. Learning multiple weights at a time: generalizing gradient descent
Gradient descent learning with multiple inputsGradient descent with multiple inputs explainedLet’s watch several steps of learningFreezing one weight: What does it do?Gradient descent learning with multiple outputsGradient descent with multiple inputs and outputsWhat do these weights learn?Visualizing weight valuesVisualizing dot products (weighted sums)Summary
Chapter 6. Building your first deep neural network: introduction to backpropagation
The streetlight problemPreparing the dataMatrices and the matrix relationshipCreating a matrix or two in PythonBuilding a neural networkLearning the whole datasetFull, batch, and stochastic gradient descentNeural networks learn correlationUp and down pressureEdge case: OverfittingEdge case: Conflicting pressureLearning indirect correlationCreating correlationStacking neural networks: A reviewBackpropagation: Long-distance error attributionBackpropagation: Why does this work?Linear vs. nonlinearWhy the neural network still doesn’t workThe secret to sometimes correlationA quick breakYour first deep neural networkBackpropagation in codeOne iteration of backpropagationPutting it all togetherWhy do deep networks matter?
Chapter 7. How to picture neural networks: in your head and on paper
It’s time to simplifyCorrelation summarizationThe previously overcomplicated visualizationThe simplified visualizationSimplifying even furtherLet’s see this network predictVisualizing using letters instead of picturesLinking the variablesEverything side by sideThe importance of visualization tools
Chapter 8. Learning signal and ignoring noise: introduction to regularization and batching
Three-layer network on MNISTWell, that was easyMemorization vs. generalizationOverfitting in neural networksWhere overfitting comes fromThe simplest regularization: Early stoppingIndustry standard regularization: DropoutWhy dropout works: Ensembling worksDropout in codeDropout evaluated on MNISTBatch gradient descentSummary
Chapter 9. Modeling probabilities and nonlinearities: activation functions
What is an activation function?Standard hidden-layer activation functionsStandard output layer activation functionsThe core issue: Inputs have similaritysoftmax computationActivation installation instructionsMultiplying delta by the slopeConverting output to slope (derivative)Upgrading the MNIST network
Chapter 10. Neural learning about edges and corners: intro to convolutional neural networks
Reusing weights in multiple placesThe convolutional layerA simple implementation in NumPySummary
Chapter 11. Neural networks that understand language: king – man + woman == ?
What does it mean to understand language?Natural language processing (NLP)Supervised NLPIMDB movie reviews datasetCapturing word correlation in input dataPredicting movie reviewsIntro to an embedding layerInterpreting the outputNeural architectureComparing word embeddingsWhat is the meaning of a neuron?Filling in the blankMeaning is derived from lossKing – Man + Woman ~= QueenWord analogiesSummary
Chapter 12. Neural networks that write like Shakespeare: recurrent layers for variable-length data
The challenge of arbitrary lengthDo comparisons really matter?The surprising power of averaged word vectorsHow is information stored in these embeddings?How does a neural network use embeddings?The limitations of bag-of-words vectorsUsing identity vectors to sum word embeddingsMatrices that change absolutely nothingLearning the transition matricesLearning to create useful sentence vectorsForward propagation in PythonHow do you backpropagate into this?Let’s train it!Setting things upForward propagation with arbitrary lengthBackpropagation with arbitrary lengthWeight update with arbitrary lengthExecution and output analysisSummary
Chapter 13. Introducing automatic optimization: let’s build a deep learning framework
What is a deep learning framework?Introduction to tensorsIntroduction to automatic gradient computation (autograd)A quick checkpointTensors that are used multiple timesUpgrading autograd to support multiuse tensorsHow does addition backpropagation work?Adding support for negationAdding support for additional functionsUsing autograd to train a neural networkAdding automatic optimizationAdding support for layer typesLayers that contain layersLoss-function layersHow to learn a frameworkNonlinearity layersThe embedding layerAdding indexing to autogradThe embedding layer (revisited)The cross-entropy layerThe recurrent neural network layerSummary
Chapter 14. Learning to write like Shakespeare: long short-term memory
Character language modelingThe need for truncated backpropagationTruncated backpropagationA sample of the outputVanishing and exploding gradientsA toy example of RNN backpropagationLong short-term memory (LSTM) cellsSome intuition about LSTM gatesThe long short-term memory layerUpgrading the character language modelTraining the LSTM character language modelTuning the LSTM character language modelSummary
Chapter 15. Deep learning on unseen data: introducing federated learning
The problem of privacy in deep learningFederated learningLearning to detect spamLet’s make it federatedHacking into federated learningSecure aggregationHomomorphic encryptionHomomorphically encrypted federated learningSummary
Chapter 16. Where to go from here: a brief guide
Congratulations!Step 1: Start learning PyTorchStep 2: Start another deep learning courseStep 3: Grab a mathy deep learning textbookStep 4: Start a blog, and teach deep learningStep 5: TwitterStep 6: Implement academic papersStep 7: Acquire access to a GPU (or many)Step 8: Get paid to practiceStep 9: Join an open source projectStep 10: Develop your local community
Index

Overview

About the Technology

Deep learning, a branch of artificial intelligence, teaches computers to learn by using neural networks, technology inspired by the human brain. Online text translation, self-driving cars, personalized product recommendations, and virtual voice assistants are just a few of the exciting modern advancements possible thanks to deep learning.

About the Book

Grokking Deep Learning teaches you to build deep learning neural networks from scratch! In his engaging style, seasoned deep learning expert Andrew Trask shows you the science under the hood, so you grok for yourself every detail of training neural networks. Using only Python and its math-supporting library, NumPy, you’ll train your own neural networks to see and understand images, translate text into different languages, and even write like Shakespeare! When you’re done, you’ll be fully prepared to move on to mastering deep learning frameworks.

What's Inside

The science behind deep learning
Building and training your own neural networks
Privacy concepts, including federated learning
Tips for continuing your pursuit of deep learning

About the Reader

For readers with high school-level math and intermediate programming skills.

About the Author

Andrew Trask is a PhD student at Oxford University and a research scientist at DeepMind. Previously, Andrew was a researcher and analytics product manager at Digital Reasoning, where he trained the world’s largest artificial neural network and helped guide the analytics roadmap for the Synthesys cognitive computing platform.

We interviewed Andrew as a part of our Six Questions series. Check it out here.

Quotes
An excellent introduction and overview of deep learning by a masterful teacher who guides, illuminates, and encourages you along the way.
- Kelvin D. Meeks, International Technology Ventures

All concepts are clearly explained with excellent visualizations and examples.
- Kalyan Reddy, ArisGlobal

Excels at navigating the reader through the introductory details of deep learning in a simple and intuitive way.
- Eremey Valetov, Lancaster University/Fermilab

Your step-by-step guide to learning AI.
- Ian Stirk, I and K Consulting

A complex topic simplified!
- Vipul Gupta, Microsoft

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617293702Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills