Deep Learning for Coders with fastai and PyTorch

Book Description

Deep learning has the reputation as an exclusive domain for math PhDs. Not so. With this book, programmers comfortable with Python will learn how to get started with deep learning right away.

Using PyTorch and the fastai deep learning library, you’ll learn how to train a model to accomplish a wide range of tasks—including computer vision, natural language processing, tabular data, and generative networks. At the same time, you’ll dig progressively into deep learning theory so that by the end of the book you’ll have a complete understanding of the math behind the library’s functions.

Publisher Resources

View/Submit Errata

Table of Contents

  1. Preface
    1. Who This Is for
    2. What Do You Need to Know
    3. What You Will Learn
    4. O’Reilly Online Learning
    5. How to Contact Us
  2. Foreword
  3. I. Deep Learning in Practice
  4. 1. Your Deep Learning Journey
    1. Deep Learning Is for Everyone
    2. Neural Networks: A Brief History
    3. Who We Are
    4. How to Learn Deep Learning
      1. Your Projects and Your Mindset
    5. The Software: PyTorch, fastai, and Jupyter (And Why It Doesn’t Matter)
    6. Your First Model
      1. Getting a GPU Deep Learning Server
      2. Running Your First Notebook
      3. What Is Machine Learning?
      4. What Is a Neural Network?
      5. Limitations Inherent to Machine Learning
      6. How Our Image Recognizer Works
      7. What Our Image Recognizer Learned
      8. Image Recognizers Can Tackle Non-Image Tasks
      9. Jargon Recap
    7. Deep Learning Is Not Just for Image Classification
    8. Validation Sets and Test Sets
      1. Use Judgment in Defining Test Sets
    9. A Choose Your Own Adventure Moment
    10. Questionnaire
      1. Further Research
  5. 2. From Model to Production
    1. The Practice of Deep Learning
      1. Starting Your Project
      2. The State of Deep Learning
      3. The Drivetrain Approach
    2. Gathering Data
    3. From Data to DataLoaders
      1. Data Augmentation
    4. Training Your Model, and Using It to Clean Your Data
    5. Turning Your Model into an Online Application
      1. Using the Model for Inference
      2. Creating a Notebook App from the Model
      3. Turning Your Notebook into a Real App
      4. Deploying Your App
    6. How to Avoid Disaster
      1. Unforeseen Consequences and Feedback Loops
    7. Get Writing!
    8. Questionnaire
      1. Further Research
  6. 3. Data Ethics
    1. Key Examples for Data Ethics
      1. Bugs and Recourse: Buggy Algorithm Used for Healthcare Benefits
      2. Feedback Loops: YouTube’s Recommendation System
      3. Bias: Professor Lantanya Sweeney “Arrested”
      4. Why Does This Matter?
    2. Integrating Machine Learning with Product Design
    3. Topics in Data Ethics
      1. Recourse and Accountability
      2. Feedback Loops
      3. Bias
    4. Addressing Different Types of Bias
      1. Humans Are Biased, so Does Algorithmic Bias Matter?
    5. Disinformation
    6. Identifying and Addressing Ethical Issues
      1. Analyze a Project You Are Working On
      2. Processes to Implement
      3. The Power of Diversity
      4. Fairness, Accountability, and Transparency
    7. Role of Policy
      1. The Effectiveness of Regulation
      2. Rights and Policy
      3. Cars: A Historical Precedent
    8. Conclusion
    9. Questionnaire
      1. Further Research:
    10. Section 1: That’s a Wrap!
  7. II. Understand fastai’s applications
  8. 4. Under the Hood: Training a Digit Classifier
    1. Pixels: The Foundations of Computer Vision
    2. First Try: Pixel Similarity
      1. NumPy Arrays and PyTorch Tensors
    3. Computing Metrics Using Broadcasting
    4. Stochastic Gradient Descent (SGD)
      1. The Gradient
      2. Stepping with a Learning Rate
      3. An End-to-End SGD Example
      4. Summarizing Gradient Descent
    5. MNIST Loss Function
      1. Sigmoid
      2. SGD and Mini-Batches
    6. Putting It All Together
      1. Creating an Optimizer
    7. Adding a Non-Linearity
    8. Jargon Recap
    9. Questionnaire
      1. Further Research
  9. 5. Image Classification
    1. From Dogs and Cats, to Pet Breeds
    2. Presizing
      1. Checking and Debugging a DataBlock
    3. Cross Entropy Loss
      1. Viewing Activations and Labels
      2. Softmax
      3. Log Likelihood
      4. Taking the log
    4. Model Interpretation
    5. Improving our Model
      1. Learning Rate Finder
      2. Unfreezing and Transfer Learning
      3. Discriminative Learning Rates
      4. Selecting the Number of Epochs
      5. Deeper Architectures
    6. Summary
    7. Questionnaire
      1. Further Research
  10. 6. Other Computer Vision Problems
    1. Multi-Label Classification
      1. The Data
      2. Constructing a Data Block
      3. Binary Cross Entropy
    2. Regression
      1. Assemble the Data
      2. Training a Model
    3. Conclusion
    4. Questionnaire
      1. Further Research
  11. 7. Training a State-of-the-Art Model
    1. Imagenette
    2. Normalization
    3. Progressive Resizing
    4. Test Time Augmentation
    5. Mixup
    6. Label Smoothing
    7. Conclusion
    8. Questionnaire
      1. Further Research
  12. 8. Collaborative Filtering Deep Dive
    1. A First Look at the Data
    2. Learning the Latent Factors
    3. Creating the DataLoaders
    4. Collaborative Filtering from Scratch
      1. Weight Decay
      2. Creating Our Own Embedding Module
    5. Interpreting Embeddings and Biases
      1. Using fastai.collab
      2. Embedding Distance
    6. Boot Strapping a Collaborative Filtering Model
    7. Deep Learning for Collaborative Filtering
    8. Conclusion
    9. Questionnaire
      1. Further Research
  13. 9. Tabular Modelling Deep Dive
    1. Categorical Embeddings
    2. Beyond Deep Learning
    3. The Dataset
      1. Kaggle Competitions
      2. Look at the Data
    4. Decision Trees
      1. Handling Dates
      2. Using TabularPandas and TabularProc
      3. Creating the Decision Tree
      4. Categorical Variables
    5. Random Forests
      1. Creating a Random Forest
      2. Out-of-Bag Error
    6. Model Interpretation
      1. Tree Variance for Prediction Confidence
      2. Feature Importance
      3. Removing Low-Importance Variables
      4. Removing Redundant Features
      5. Partial Dependence
      6. Data Leakage
      7. Tree Interpreter
    7. Extrapolation and Neural Networks
      1. The Extrapolation Problem
      2. Finding out of Domain Data
      3. Using a Neural Network
      4. Ensembling
      5. Boosting
      6. Combining Embeddings with Other Methods
    8. Conclusion: Our Advice for Tabular Modeling
    9. Questionnaire
      1. Further Research
  14. 10. NLP Deep Dive: RNNs
    1. Text Preprocessing
      1. Tokenization
      2. Word Tokenization with fastai
      3. Subword Tokenization
      4. Numericalization with fastai
      5. Putting Our Texts Into Batches for a Language Model
    2. Training a Text Classifier
      1. Language Model Using DataBlock
      2. Fine Tuning the Language Model
      3. Saving and Loading Models
      4. Text Generation
      5. Creating the Classifier DataLoaders
      6. Fine Tuning the Classifier
    3. Disinformation and Language Models
    4. Conclusion
    5. Questionnaire
      1. Further Research
  15. 11. Data Munging With fastai’s mid-Level API
    1. Going Deeper into fastai’s Layered API
      1. Transforms
      2. Writing Your Own Transform
      3. Pipeline
    2. TfmdLists and Datasets: Transformed Collections
      1. TfmdLists
      2. Datasets
    3. Applying the mid-Tier Data API: SiamesePair
    4. Conclusion
    5. Questionnaire
      1. Further Research
    6. Becoming a Deep Learning Practitioner
  16. III. Foundations of Deep Learning
  17. 12. A Language Model from Scratch
    1. The Data
    2. Our First Language Model from Scratch
      1. Our Language Model in PyTorch
      2. Our First Recurrent Neural Network
    3. Improving the RNN
      1. Maintaining the State of an RNN
      2. Creating More Signal
    4. Multilayer RNNs
    5. The Model
      1. Exploding or Disappearing Activations
    6. LSTM
      1. Building an LSTM from Scratch
      2. Training a Language Model Using LSTMs
    7. Regularizing an LSTM
      1. Dropout
      2. AR and TAR Regularization
      3. Training a Weight-Tied Regularized LSTM
    8. Conclusion
    9. Questionnaire
      1. Further Research
  18. 13. Convolutional Neural Networks
    1. The Magic of Convolutions
      1. Mapping a Convolution kernel
      2. Convolutions in PyTorch
      3. Strides and Padding
      4. Understanding the Convolution Equations
    2. Our First Convolutional Neural Network
      1. Creating the CNN
      2. Understanding Convolution Arithmetic
      3. Receptive Fields
      4. A Note about Twitter
    3. Colour Images
    4. Improving Training Stability
      1. A Simple Baseline
      2. Increase Batch Size
      3. 1cycle Training
      4. Batch Normalization
    5. Conclusions
    6. Questionnaire
      1. Further Research
  19. 14. Resnets
    1. Going Back to Imagenette
    2. Building a Modern CNN: ResNet
      1. Skip-Connections
      2. A State-of-the-Art ResNet
      3. Bottleneck Layers
    3. Conclusion
    4. Questionnaire
      1. Further Research
  20. 15. Application Architectures Deep Dive
    1. Computer Vision
      1. cnn_learner
      2. unet_learner
      3. A Siamese Network
    2. Natural Language Processing
    3. Tabular
    4. Wrapping up Architectures
    5. Questionnaire
      1. Further Research
  21. 16. The Training Process
    1. Let’s Start with SGD
    2. A Generic Optimizer
    3. Momentum
    4. RMSProp
    5. Adam
    6. Decoupled weight_decay
    7. Callbacks
      1. Creating a Callback
      2. Callback Ordering and Exceptions
    8. Conclusion
    9. Questionnaire
      1. Further Research
    10. Foundations of Deep Learning: Wrap Up
  22. IV. Deep Learning from scratch
  23. 17. A Neural Net from the Foundations
    1. A Neural Net Layer from Scratch
      1. Modeling a Neuron
      2. Matrix Multiplication from Scratch
    2. Elementwise Arithmetic
      1. Broadcasting
      2. Einstein Summation
    3. The Forward and Backward Passes
      1. Defining and Initializing a Layer
      2. Gradients and Backward Pass
      3. Refactor the Model
      4. Going to PyTorch
    4. Conclusion
    5. Questionnaire
      1. Further Research
  24. 18. CNN Interpretation with CAM
    1. CAM and Hooks
    2. Gradient CAM
    3. Conclusion
    4. Questionnaire
      1. Further Research
  25. 19. fastai Learner from Scratch
    1. Data
      1. Dataset
    2. Module and Parameter
      1. Simple CNN
    3. Loss
    4. Learner
      1. Callbacks
      2. Scheduling the Learning Rate
    5. Conclusion
    6. Questionnaire
      1. Further Research
  26. 20. Concluding Thoughts
  27. A. Creating a Blog
    1. Blogging with GitHub Pages
      1. Creating the Repository
      2. Setting up Your Homepage
      3. Creating Posts
      4. Synchronizing GitHub and Your Computer
      5. Jupyter for Blogging
  28. B. Appendix: Data Project Checklist
    1. Organizational
      1. Data scientists
      2. Strategy
    2. Data
    3. Analytics
    4. Implementation
    5. Maintenance
    6. Constraints

Product Information

  • Title: Deep Learning for Coders with fastai and PyTorch
  • Author(s): Jeremy Howard, Sylvain Gugger
  • Release date: July 2020
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781492045526