book

Deep Learning for Coders with fastai and PyTorch

by Jeremy Howard, Sylvain Gugger

July 2020

Intermediate to advanced

621 pages

16h 47m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Preface
Who This Book Is ForWhat You Need to KnowWhat You Will LearnO’Reilly Online LearningHow to Contact Us
Foreword
I. Deep Learning in Practice
1. Your Deep Learning Journey
Deep Learning Is for EveryoneNeural Networks: A Brief HistoryWho We AreHow to Learn Deep LearningYour Projects and Your MindsetThe Software: PyTorch, fastai, and Jupyter (And Why It Doesn’t Matter)Your First ModelGetting a GPU Deep Learning ServerRunning Your First NotebookWhat Is Machine Learning?What Is a Neural Network?A Bit of Deep Learning JargonLimitations Inherent to Machine LearningHow Our Image Recognizer WorksWhat Our Image Recognizer LearnedImage Recognizers Can Tackle Non-Image TasksJargon RecapDeep Learning Is Not Just for Image ClassificationValidation Sets and Test SetsUse Judgment in Defining Test SetsA Choose Your Own Adventure MomentQuestionnaireFurther Research
2. From Model to Production
The Practice of Deep LearningStarting Your ProjectThe State of Deep LearningThe Drivetrain ApproachGathering DataFrom Data to DataLoadersData AugmentationTraining Your Model, and Using It to Clean Your DataTurning Your Model into an Online ApplicationUsing the Model for InferenceCreating a Notebook App from the ModelTurning Your Notebook into a Real AppDeploying Your AppHow to Avoid DisasterUnforeseen Consequences and Feedback LoopsGet Writing!QuestionnaireFurther Research
3. Data Ethics
Key Examples for Data EthicsBugs and Recourse: Buggy Algorithm Used for Healthcare BenefitsFeedback Loops: YouTube’s Recommendation SystemBias: Professor Latanya Sweeney “Arrested”Why Does This Matter?Integrating Machine Learning with Product DesignTopics in Data EthicsRecourse and AccountabilityFeedback LoopsBiasDisinformationIdentifying and Addressing Ethical IssuesAnalyze a Project You Are Working OnProcesses to ImplementThe Power of DiversityFairness, Accountability, and TransparencyRole of PolicyThe Effectiveness of RegulationRights and PolicyCars: A Historical PrecedentConclusionQuestionnaireFurther ResearchDeep Learning in Practice: That’s a Wrap!
II. Understanding fastai’s Applications
4. Under the Hood: Training a Digit Classifier
Pixels: The Foundations of Computer VisionFirst Try: Pixel SimilarityNumPy Arrays and PyTorch TensorsComputing Metrics Using BroadcastingStochastic Gradient DescentCalculating GradientsStepping with a Learning RateAn End-to-End SGD ExampleSummarizing Gradient DescentThe MNIST Loss FunctionSigmoidSGD and Mini-BatchesPutting It All TogetherCreating an OptimizerAdding a NonlinearityGoing DeeperJargon RecapQuestionnaireFurther Research
5. Image Classification
From Dogs and Cats to Pet BreedsPresizingChecking and Debugging a DataBlockCross-Entropy LossViewing Activations and LabelsSoftmaxLog LikelihoodTaking the logModel InterpretationImproving Our ModelThe Learning Rate FinderUnfreezing and Transfer LearningDiscriminative Learning RatesSelecting the Number of EpochsDeeper ArchitecturesConclusionQuestionnaireFurther Research
6. Other Computer Vision Problems
Multi-Label ClassificationThe DataConstructing a DataBlockBinary Cross EntropyRegressionAssembling the DataTraining a ModelConclusionQuestionnaireFurther Research

7. Training a State-of-the-Art Model
ImagenetteNormalizationProgressive ResizingTest Time AugmentationMixupLabel SmoothingConclusionQuestionnaireFurther Research
8. Collaborative Filtering Deep Dive
A First Look at the DataLearning the Latent FactorsCreating the DataLoadersCollaborative Filtering from ScratchWeight DecayCreating Our Own Embedding ModuleInterpreting Embeddings and BiasesUsing fastai.collabEmbedding DistanceBootstrapping a Collaborative Filtering ModelDeep Learning for Collaborative FilteringConclusionQuestionnaireFurther Research
9. Tabular Modeling Deep Dive
Categorical EmbeddingsBeyond Deep LearningThe DatasetKaggle CompetitionsLook at the DataDecision TreesHandling DatesUsing TabularPandas and TabularProcCreating the Decision TreeCategorical VariablesRandom ForestsCreating a Random ForestOut-of-Bag ErrorModel InterpretationTree Variance for Prediction ConfidenceFeature ImportanceRemoving Low-Importance VariablesRemoving Redundant FeaturesPartial DependenceData LeakageTree InterpreterExtrapolation and Neural NetworksThe Extrapolation ProblemFinding Out-of-Domain DataUsing a Neural NetworkEnsemblingBoostingCombining Embeddings with Other MethodsConclusionQuestionnaireFurther Research
10. NLP Deep Dive: RNNs
Text PreprocessingTokenizationWord Tokenization with fastaiSubword TokenizationNumericalization with fastaiPutting Our Texts into Batches for a Language ModelTraining a Text ClassifierLanguage Model Using DataBlockFine-Tuning the Language ModelSaving and Loading ModelsText GenerationCreating the Classifier DataLoadersFine-Tuning the ClassifierDisinformation and Language ModelsConclusionQuestionnaireFurther Research
11. Data Munging with fastai’s Mid-Level API
Going Deeper into fastai’s Layered APITransformsWriting Your Own TransformPipelineTfmdLists and Datasets: Transformed CollectionsTfmdListsDatasetsApplying the Mid-Level Data API: SiamesePairConclusionQuestionnaireFurther ResearchUnderstanding fastai’s Applications: Wrap Up
III. Foundations of Deep Learning
12. A Language Model from Scratch
The DataOur First Language Model from ScratchOur Language Model in PyTorchOur First Recurrent Neural NetworkImproving the RNNMaintaining the State of an RNNCreating More SignalMultilayer RNNsThe ModelExploding or Disappearing ActivationsLSTMBuilding an LSTM from ScratchTraining a Language Model Using LSTMsRegularizing an LSTMDropoutActivation Regularization and Temporal Activation RegularizationTraining a Weight-Tied Regularized LSTMConclusionQuestionnaireFurther Research
13. Convolutional Neural Networks
The Magic of ConvolutionsMapping a Convolutional KernelConvolutions in PyTorchStrides and PaddingUnderstanding the Convolution EquationsOur First Convolutional Neural NetworkCreating the CNNUnderstanding Convolution ArithmeticReceptive FieldsA Note About TwitterColor ImagesImproving Training StabilityA Simple BaselineIncrease Batch Size1cycle TrainingBatch NormalizationConclusionQuestionnaireFurther Research
14. ResNets
Going Back to ImagenetteBuilding a Modern CNN: ResNetSkip ConnectionsA State-of-the-Art ResNetBottleneck LayersConclusionQuestionnaireFurther Research
15. Application Architectures Deep Dive
Computer Visioncnn_learnerunet_learnerA Siamese NetworkNatural Language ProcessingTabularConclusionQuestionnaireFurther Research
16. The Training Process
Establishing a BaselineA Generic OptimizerMomentumRMSPropAdamDecoupled Weight DecayCallbacksCreating a CallbackCallback Ordering and ExceptionsConclusionQuestionnaireFurther ResearchFoundations of Deep Learning: Wrap Up
IV. Deep Learning from Scratch
17. A Neural Net from the Foundations
Building a Neural Net Layer from ScratchModeling a NeuronMatrix Multiplication from ScratchElementwise ArithmeticBroadcastingEinstein SummationThe Forward and Backward PassesDefining and Initializing a LayerGradients and the Backward PassRefactoring the ModelGoing to PyTorchConclusionQuestionnaireFurther Research
18. CNN Interpretation with CAM
CAM and HooksGradient CAMConclusionQuestionnaireFurther Research
19. A fastai Learner from Scratch
DataDatasetModule and ParameterSimple CNNLossLearnerCallbacksScheduling the Learning RateConclusionQuestionnaireFurther Research
20. Concluding Thoughts
A. Creating a Blog
Blogging with GitHub PagesCreating the RepositorySetting Up Your Home PageCreating PostsSynchronizing GitHub and Your ComputerJupyter for Blogging
B. Data Project Checklist
Data ScientistsStrategyDataAnalyticsImplementationMaintenanceConstraints
Index

Content preview from Deep Learning for Coders with fastai and PyTorch

Appendix B. Data Project Checklist

There’s a lot more to creating useful data projects than just training an accurate model! When Jeremy used to do consulting, he’d always seek to understand an organization’s context for developing data projects based on the following considerations, summarized in Figure B-1:

Strategy: What is the organization trying to do (objective), and what can it change to do it better (levers)?
Data: Is the organization capturing the necessary data and making it available?
Analytics: What kinds of insights would be useful to the organization?
Implementation: What organizational capabilities are available?
Maintenance: What systems are in place to track changes in the operational environment?
Constraints: What constraints need to be considered in each of the preceding areas?

He developed a questionnaire that he had clients fill out before a project started, and then throughout the project he’d help them refine their answers. This questionnaire is based on decades of projects across many industries, including agriculture, mining, banking, brewing, telecoms, retail, and more.

Before we go through the analytics value chain, the first part of the questionnaire has to do with the most important employees for your data project: data scientists.

Data Scientists

Data scientists should have a clear path to becoming senior ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Build a Large Language Model (From Scratch)

Publisher Resources

ISBN: 9781492045519Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Deep Learning for Coders with fastai and PyTorch

by Jeremy Howard, Sylvain Gugger

Appendix B. Data Project Checklist

Figure B-1. The analytics value chain

Data Scientists

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.