Programming Machine Learning

A Regularization Toolbox

Just like tuning hyperparameters, reducing overfitting is more art than science. Besides L1 and L2, there are many other regularization methods you can use. Here’s an overview of some of them.

The most fundamental regularization technique is also the first one you should reach for: make the overfitting network smaller. After all, overfitting happens because the system is too clever for the data it’s learning. Smaller networks are not as clever as big networks. Try reducing the number of hidden nodes, or even removing a few layers. You’ll have a go at this approach in the chapter’s closing exercise.

Instead of simplifying the model, you can also reduce overfitting by simplifying the data—that is, removing a few input ...