Programming PyTorch for Deep Learning

Table of contents

1. Preface
   1. Deep Learning in the World Today
   2. But What Is Deep Learning Exactly, and Do I Need a PhD to Understand It?
   3. PyTorch
      1. What About TensorFlow?
   4. Conventions Used in This Book
   5. Using Code Examples
   6. O’Reilly Online Learning
   7. How to Contact Us
   8. Acknowledgments
2. 1. Getting Started with PyTorch
   1. Building a Custom Deep Learning Machine
      1. GPU
      2. CPU/Motherboard
      3. RAM
      4. Storage
   2. Deep Learning in the Cloud
      1. Google Colaboratory
      2. Cloud Providers
      3. Which Cloud Provider Should I Use?
   3. Using Jupyter Notebook
   4. Installing PyTorch from Scratch
      1. Download CUDA
      2. Anaconda
      3. Finally, PyTorch! (and Jupyter Notebook)
   5. Tensors
      1. Tensor Operations
      2. Tensor Broadcasting
   6. Conclusion
   7. Further Reading
3. 2. Image Classification with PyTorch
   1. Our Classification Problem
   2. Traditional Challenges
      1. But First, Data
      2. PyTorch and Data Loaders
      3. Building a Training Dataset
      4. Building Validation and Test Datasets
   3. Finally, a Neural Network!
      1. Activation Functions
      2. Creating a Network
      3. Loss Functions
      4. Optimizing
   4. Training
      1. Making It Work on the GPU
   5. Putting It All Together
      1. Making Predictions
      2. Model Saving
   6. Conclusion
   7. Further Reading
4. 3. Convolutional Neural Networks
   1. Our First Convolutional Model
      1. Convolutions
      2. Pooling
      3. Dropout
   2. History of CNN Architectures
      1. AlexNet
      2. Inception/GoogLeNet
      3. VGG
      4. ResNet
      5. Other Architectures Are Available!
   3. Using Pretrained Models in PyTorch
      1. Examining a Model’s Structure
      2. BatchNorm
      3. Which Model Should You Use?
   4. One-Stop Shopping for Models: PyTorch Hub
   5. Conclusion
   6. Further Reading
5. 4. Transfer Learning and Other Tricks
   1. Transfer Learning with ResNet
   2. Finding That Learning Rate
   3. Differential Learning Rates
   4. Data Augmentation
      1. Torchvision Transforms
      2. Color Spaces and Lambda Transforms
      3. Custom Transform Classes
      4. Start Small and Get Bigger!
   5. Ensembles
   6. Conclusion
   7. Further Reading
6. 5. Text Classification
   1. Recurrent Neural Networks
   2. Long Short-Term Memory Networks
      1. Gated Recurrent Units
      2. biLSTM
   3. Embeddings
   4. torchtext
      1. Getting Our Data: Tweets!
      2. Defining Fields
      3. Building a Vocabulary
      4. Creating Our Model
      5. Updating the Training Loop
      6. Classifying Tweets
   5. Data Augmentation
      1. Random Insertion
      2. Random Deletion
      3. Random Swap
      4. Back Translation
      5. Augmentation and torchtext
      6. Transfer Learning?
   6. Conclusion
   7. Further Reading
7. 6. A Journey into Sound
   1. Sound
   2. The ESC-50 Dataset
      1. Obtaining the Dataset
      2. Playing Audio in Jupyter
   3. Exploring ESC-50
      1. SoX and LibROSA
      2. torchaudio
      3. Building an ESC-50 Dataset
   4. A CNN Model for ESC-50
   5. This Frequency Is My Universe
      1. Mel Spectrograms
      2. A New Dataset
      3. A Wild ResNet Appears
      4. Finding a Learning Rate
   6. Audio Data Augmentation
      1. torchaudio Transforms
      2. SoX Effect Chains
      3. SpecAugment
   7. Further Experiments
   8. Conclusion
   9. Further Reading
8. 7. Debugging PyTorch Models
   1. It’s 3 a.m. What Is Your Data Doing?
   2. TensorBoard
      1. Installing TensorBoard
      2. Sending Data to TensorBoard
      3. PyTorch Hooks
      4. Plotting Mean and Standard Deviation
      5. Class Activation Mapping
   3. Flame Graphs
      1. Installing py-spy
      2. Reading Flame Graphs
      3. Fixing a Slow Transformation
   4. Debugging GPU Issues
      1. Checking Your GPU
      2. Gradient Checkpointing
   5. Conclusion
   6. Further Reading
9. 8. PyTorch in Production
   1. Model Serving
      1. Building a Flask Service
      2. Setting Up the Model Parameters
      3. Building the Docker Container
      4. Local Versus Cloud Storage
      5. Logging and Telemetry
   2. Deploying on Kubernetes
      1. Setting Up on Google Kubernetes Engine
      2. Creating a k8s Cluster
      3. Scaling Services
      4. Updates and Cleaning Up
   3. TorchScript
      1. Tracing
      2. Scripting
      3. TorchScript Limitations
   4. Working with libTorch
      1. Obtaining libTorch and Hello World
      2. Importing a TorchScript Model
   5. Conclusion
   6. Further Reading
10. 9. PyTorch in the Wild
    1. Data Augmentation: Mixed and Smoothed
       1. mixup
       2. Label Smoothing
    2. Computer, Enhance!
       1. Introduction to Super-Resolution
       2. An Introduction to GANs
       3. The Forger and the Critic
       4. Training a GAN
       5. The Dangers of Mode Collapse
       6. ESRGAN
    3. Further Adventures in Image Detection
       1. Object Detection
       2. Faster R-CNN and Mask R-CNN
    4. Adversarial Samples
       1. Black-Box Attacks
       2. Defending Against Adversarial Attacks
    5. More Than Meets the Eye: The Transformer Architecture
       1. Paying Attention
       2. Attention Is All You Need
       3. BERT
       4. FastBERT
       5. GPT-2
       6. Generating Text with GPT-2
       7. ULMFiT
       8. What to Use?
    6. Conclusion
    7. Further Reading
11. Index