Deep Learning Patterns and Practices

4 Training fundamentals

This chapter covers

• Forward feeding and backward propagation
• Splitting datasets and preprocessing data
• Using validation data to monitor overfitting
• Using checkpointing and early stopping for more-economical training
• Using hyperparameters versus model parameters
• Training for invariance to location and scale
• Assembling and accessing on-disk datasets
• Saving and then restoring a trained model

This chapter covers the fundamentals of training a model. Prior to 2019, the majority of models were trained according to this set of fundamental steps. Consider this chapter as a foundation.

In this chapter, we cover methods, techniques, and best practices developed over time by experimentation and trial and error. We will start ...