Deep Learning Patterns and Practices

10 Hyperparameter tuning

This chapter covers

• Initializing the weights in a model prior to warm-up training
• Doing hyperparameter search manually and automatically
• Constructing a learning rate scheduler for training a model
• Regularizing a model during training

Hyperparameter tuning is the process of finding the optimal settings of the training hyperparameters, so that we minimize the training time and maximize the test accuracy. Usually, these two objectives can’t be fully optimized. If we minimize the training time, we likely will not achieve the best accuracy. Likewise, if we maximize the test accuracy, we likely will need longer to train.

Tuning is finding the combination of hyperparameter settings that meet your targets for the objectives. ...