Programming Machine Learning

What You Just Learned

This chapter was all about overfitting. Overfitting happens when the system learns the statistical noise in the training data, and fails to generalize that knowledge to new data. The more powerful a supervised learning system is, the more likely it is to overfit. Deep neural networks are very powerful, so they’re very prone to overfitting.

You can reduce overfitting by “smoothing out” the neural network’s model, so that it follows the general shape of the data instead of tracking every noisy fluctuation. That idea is called regularization. In this chapter, we looked at a few regularization techniques:

• L1 and L2 regularization
• Simplifying the model
• Simplifying the training data
• Early stopping
• Increasing the learning rate

To understand ...