Deep Learning Patterns and Practices

Deep Learning Patterns and Practices

by Andrew Ferlitsch
Released September 2021
Publisher(s): Manning Publications
ISBN: 9781617298264

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Discover best practices, reproducible architectures, and design patterns to help guide deep learning models from the lab into production.

In Deep Learning Patterns and Practices you will learn:

  • Internal functioning of modern convolutional neural networks
  • Procedural reuse design pattern for CNN architectures
  • Models for mobile and IoT devices
  • Assembling large-scale model deployments
  • Optimizing hyperparameter tuning
  • Migrating a model to a production environment

The big challenge of deep learning lies in taking cutting-edge technologies from R&D labs through to production. Deep Learning Patterns and Practices is here to help. This unique guide lays out the latest deep learning insights from author Andrew Ferlitsch’s work with Google Cloud AI. In it, you'll find deep learning models presented in a unique new way: as extendable design patterns you can easily plug-and-play into your software projects. Each valuable technique is presented in a way that's easy to understand and filled with accessible diagrams and code samples.

About the Technology
Discover best practices, design patterns, and reproducible architectures that will guide your deep learning projects from the lab into production. This awesome book collects and illuminates the most relevant insights from a decade of real world deep learning experience. You’ll build your skills and confidence with each interesting example.

About the Book
Deep Learning Patterns and Practices is a deep dive into building successful deep learning applications. You’ll save hours of trial-and-error by applying proven patterns and practices to your own projects. Tested code samples, real-world examples, and a brilliant narrative style make even complex concepts simple and engaging. Along the way, you’ll get tips for deploying, testing, and maintaining your projects.

What's Inside
  • Modern convolutional neural networks
  • Design pattern for CNN architectures
  • Models for mobile and IoT devices
  • Large-scale model deployments
  • Examples for computer vision


About the Reader
For machine learning engineers familiar with Python and deep learning.

About the Author
Andrew Ferlitsch is an expert on computer vision, deep learning, and operationalizing ML in production at Google Cloud AI Developer Relations.

Quotes
One of the best deep learning books I have read.
- Muhammad Sohaib Arif, Tek Systems

A must-read for anyone trying to understand the ins and outs of deep learning and best practices for building a machine learning pipeline.
- Ariel Gamiño, GLG

Really good for the modern deep learning professional. Suggested for those who want to understand what’s under the hood.
- Simone Sguazza, University of Applied Sciences and Arts of Southern Switzerland

Whether you are new to AI or have some knowledge of it, this will be your companion on the path to expertise.
- Eros Pedrini, everis

Table of contents

  1. Deep Learning Patterns and Practices
  2. Copyright
  3. brief contents
  4. contents
  5. front matter
    1. preface
    2. acknowledgments
    3. about this book
      1. Who should read this book
      2. How this book is organized: A roadmap
      3. About the code
      4. liveBook discussion forum
      5. Other online resources
    4. about the author
    5. about the cover illustration
  6. Part 1. Deep learning fundamentals
  7. 1 Designing modern machine learning
    1. 1.1 A focus on adaptability
      1. 1.1.1 Computer vision leading the way
      2. 1.1.2 Beyond computer vision: NLP, NLU, structured data
    2. 1.2 The evolution in machine learning approaches
      1. 1.2.1 Classical AI vs. narrow AI
      2. 1.2.2 Next steps in computer learning
    3. 1.3 The benefits of design patterns
    4. Summary
  8. 2 Deep neural networks
    1. 2.1 Neural network basics
      1. 2.1.1 Input layer
      2. 2.1.2 Deep neural networks
      3. 2.1.3 Feed-forward networks
      4. 2.1.4 Sequential API method
      5. 2.1.5 Functional API method
      6. 2.1.6 Input shape vs. input layer
      7. 2.1.7 Dense layer
      8. 2.1.8 Activation functions
      9. 2.1.9 Shorthand syntax
      10. 2.1.10 Improving accuracy with an optimizer
    2. 2.2 DNN binary classifier
    3. 2.3 DNN multiclass classifier
    4. 2.4 DNN multilabel multiclass classifier
    5. 2.5 Simple image classifier
      1. 2.5.1 Flattening
      2. 2.5.2 Overfitting and dropout
    6. Summary
  9. 3 Convolutional and residual neural networks
    1. 3.1 Convolutional neural networks
      1. 3.1.1 Why we use a CNN over a DNN for image models
      2. 3.1.2 Downsampling (resizing)
      3. 3.1.3 Feature detection
      4. 3.1.4 Pooling
      5. 3.1.5 Flattening
    2. 3.2 The ConvNet design for a CNN
    3. 3.3 VGG networks
    4. 3.4 ResNet networks
      1. 3.4.1 Architecture
      2. 3.4.2 Batch normalization
      3. 3.4.3 ResNet50
    5. Summary
  10. 4 Training fundamentals
    1. 4.1 Forward feeding and backward propagation
      1. 4.1.1 Feeding
      2. 4.1.2 Backward propagation
    2. 4.2 Dataset splitting
      1. 4.2.1 Training and test sets
      2. 4.2.2 One-hot encoding
    3. 4.3 Data normalization
      1. 4.3.1 Normalization
      2. 4.3.2 Standardization
    4. 4.4 Validation and overfitting
      1. 4.4.1 Validation
      2. 4.4.2 Loss monitoring
      3. 4.4.3 Going deeper with layers
    5. 4.5 Convergence
    6. 4.6 Checkpointing and early stopping
      1. 4.6.1 Checkpointing
      2. 4.6.2 Early stopping
    7. 4.7 Hyperparameters
      1. 4.7.1 Epochs
      2. 4.7.2 Steps
      3. 4.7.3 Batch size
      4. 4.7.4 Learning rate
    8. 4.8 Invariance
      1. 4.8.1 Translational invariance
      2. 4.8.2 Scale invariance
      3. 4.8.3 TF.Keras ImageDataGenerator
    9. 4.9 Raw (disk) datasets
      1. 4.9.1 Directory structure
      2. 4.9.2 CSV file
      3. 4.9.3 JSON file
      4. 4.9.4 Reading images
      5. 4.9.5 Resizing
    10. 4.10 Model save/restore
      1. 4.10.1 Save
      2. 4.10.2 Restore
    11. Summary
  11. Part 2. Basic design pattern
  12. 5 Procedural design pattern
    1. 5.1 Basic neural network architecture
    2. 5.2 Stem component
      1. 5.2.1 VGG
      2. 5.2.2 ResNet
      3. 5.2.3 ResNeXt
      4. 5.2.4 Xception
    3. 5.3 Pre-stem
    4. 5.4 Learner component
      1. 5.4.1 ResNet
      2. 5.4.2 DenseNet
    5. 5.5 Task component
      1. 5.5.1 ResNet
      2. 5.5.2 Multilayer output
      3. 5.5.3 SqueezeNet
    6. 5.6 Beyond computer vision: NLP
      1. 5.6.1 Natural-language understanding
      2. 5.6.2 Transformer architecture
    7. Summary
  13. 6 Wide convolutional neural networks
    1. 6.1 Inception v1
      1. 6.1.1 Naive inception module
      2. 6.1.2 Inception v1 module
      3. 6.1.3 Stem
      4. 6.1.4 Learner
      5. 6.1.5 Auxiliary classifiers
      6. 6.1.6 Classifier
    2. 6.2 Inception v2: Factoring convolutions
    3. 6.3 Inception v3: Architecture redesign
      1. 6.3.1 Inception groups and blocks
      2. 6.3.2 Normal convolution
      3. 6.3.3 Spatial separable convolution
      4. 6.3.4 Stem redesign and implementation
      5. 6.3.5 Auxiliary classifier
    4. 6.4 ResNeXt: Wide residual neural networks
      1. 6.4.1 ResNeXt block
      2. 6.4.2 ResNeXt architecture
    5. 6.5 Wide residual network
      1. 6.5.1 WRN-50-2 architecture
      2. 6.5.2 Wide residual block
    6. 6.6 Beyond computer vision: Structured data
    7. Summary
  14. 7 Alternative connectivity patterns
    1. 7.1 DenseNet: Densely connected convolutional neural network
      1. 7.1.1 Dense group
      2. 7.1.2 Dense block
      3. 7.1.3 DenseNet macro-architecture
      4. 7.1.4 Dense transition block
    2. 7.2 Xception: Extreme Inception
      1. 7.2.1 Xception architecture
      2. 7.2.2 Entry flow of Xception
      3. 7.2.3 Middle flow of Xception
      4. 7.2.4 Exit flow of Xception
      5. 7.2.5 Depthwise separable convolution
      6. 7.2.6 Depthwise convolution
      7. 7.2.7 Pointwise convolution
    3. 7.3 SE-Net: Squeeze and excitation
      1. 7.3.1 Architecture of SE-Net
      2. 7.3.2 Group and block of SE-Net
      3. 7.3.3 SE link
    4. Summary
  15. 8 Mobile convolutional neural networks
    1. 8.1 MobileNet v1
      1. 8.1.1 Architecture
      2. 8.1.2 Width multiplier
      3. 8.1.3 Resolution multiplier
      4. 8.1.4 Stem
      5. 8.1.5 Learner
      6. 8.1.6 Classifier
    2. 8.2 MobileNet v2
      1. 8.2.1 Architecture
      2. 8.2.2 Stem
      3. 8.2.3 Learner
      4. 8.2.4 Classifier
    3. 8.3 SqueezeNet
      1. 8.3.1 Architecture
      2. 8.3.2 Stem
      3. 8.3.3 Learner
      4. 8.3.4 Classifier
      5. 8.3.5 Bypass connections
    4. 8.4 ShuffleNet v1
      1. 8.4.1 Architecture
      2. 8.4.2 Stem
      3. 8.4.3 Learner
    5. 8.5 Deployment
      1. 8.5.1 Quantization
      2. 8.5.2 TF Lite conversion and prediction
    6. Summary
  16. 9 Autoencoders
    1. 9.1 Deep neural network autoencoders
      1. 9.1.1 Autoencoder architecture
      2. 9.1.2 Encoder
      3. 9.1.3 Decoder
      4. 9.1.4 Training
    2. 9.2 Convolutional autoencoders
      1. 9.2.1 Architecture
      2. 9.2.2 Encoder
      3. 9.2.3 Decoder
    3. 9.3 Sparse autoencoders
    4. 9.4 Denoising autoencoders
    5. 9.5 Super-resolution
      1. 9.5.1 Pre-upsampling SR
      2. 9.5.2 Post-upsampling SR
    6. 9.6 Pretext tasks
    7. 9.7 Beyond computer vision: sequence to sequence
    8. Summary
  17. Part 3. Working with pipelines
  18. 10 Hyperparameter tuning
    1. 10.1 Weight initialization
      1. 10.1.1 Weight distributions
      2. 10.1.2 Lottery hypothesis
      3. 10.1.3 Warm-up (numerical stability)
    2. 10.2 Hyperparameter search fundamentals
      1. 10.2.1 Manual method for hyperparameter search
      2. 10.2.2 Grid search
      3. 10.2.3 Random search
      4. 10.2.4 KerasTuner
    3. 10.3 Learning rate scheduler
      1. 10.3.1 Keras decay parameter
      2. 10.3.2 Keras learning rate scheduler
      3. 10.3.3 Ramp
      4. 10.3.4 Constant step
      5. 10.3.5 Cosine annealing
    4. 10.4 Regularization
      1. 10.4.1 Weight regularization
      2. 10.4.2 Label smoothing
    5. 10.5 Beyond computer vision
    6. Summary
  19. 11 Transfer learning
    1. 11.1 TF.Keras prebuilt models
      1. 11.1.1 Base model
      2. 11.1.2 Pretrained ImageNet models for prediction
      3. 11.1.3 New classifier
    2. 11.2 TF Hub prebuilt models
      1. 11.2.1 Using TF Hub pretrained models
      2. 11.2.2 New classifier
    3. 11.3 Transfer learning between domains
      1. 11.3.1 Similar tasks
      2. 11.3.2 Distinct tasks
      3. 11.3.3 Domain-specific weights
      4. 11.3.4 Domain transfer weight initialization
      5. 11.3.5 Negative transfer
    4. 11.4 Beyond computer vision
    5. Summary
  20. 12 Data distributions
    1. 12.1 Distribution types
      1. 12.1.1 Population distribution
      2. 12.1.2 Sampling distribution
      3. 12.1.3 Subpopulation distribution
    2. 12.2 Out of distribution
      1. 12.2.1 The MNIST curated dataset
      2. 12.2.2 Setting up the environment
      3. 12.2.3 The challenge (“in the wild”)
      4. 12.2.4 Training as a DNN
      5. 12.2.5 Training as a CNN
      6. 12.2.6 Image augmentation
      7. 12.2.7 Final test
    3. Summary
  21. 13 Data pipeline
    1. 13.1 Data formats and storage
      1. 13.1.1 Compressed and raw-image formats
      2. 13.1.2 HDF5 format
      3. 13.1.3 DICOM format
      4. 13.1.4 TFRecord format
    2. 13.2 Data feeding
      1. 13.2.1 NumPy
      2. 13.2.2 TFRecord
    3. 13.3 Data preprocessing
      1. 13.3.1 Preprocessing with a pre-stem
      2. 13.3.2 Preprocessing with TF Extended
    4. 13.4 Data augmentation
      1. 13.4.1 Invariance
      2. 13.4.2 Augmentation with tf.data
      3. 13.4.3 Pre-stem
    5. Summary
  22. 14 Training and deployment pipeline
    1. 14.1 Model feeding
      1. 14.1.1 Model feeding with tf.data.Dataset
      2. 14.1.2 Distributed feeding with tf.Strategy
      3. 14.1.3 Model feeding with TFX
    2. 14.2 Training schedulers
      1. 14.2.1 Pipeline versioning
      2. 14.2.2 Metadata
      3. 14.2.3 History
    3. 14.3 Model evaluations
      1. 14.3.1 Candidate vs. blessed model
      2. 14.3.2 TFX evaluation
    4. 14.4 Serving predictions
      1. 14.4.1 On-demand (live) serving
      2. 14.4.2 Batch prediction
      3. 14.4.3 TFX pipeline components for deployment
      4. 14.4.4 A/B testing
      5. 14.4.5 Load balancing
      6. 14.4.6 Continuous evaluation
    5. 14.5 Evolution in production pipeline design
      1. 14.5.1 Machine learning as a pipeline
      2. 14.5.2 Machine learning as a CI/CD production process
      3. 14.5.3 Model amalgamation in production
    6. Summary
  23. index

Product information

  • Title: Deep Learning Patterns and Practices
  • Author(s): Andrew Ferlitsch
  • Release date: September 2021
  • Publisher(s): Manning Publications
  • ISBN: 9781617298264