Programming Machine Learning

Regularizing the Model

You learned three important concepts in the previous section:

• Powerful neural networks tend to overfit.
• Simple neural networks tend to underfit.
• You should strike a balance between the two.

Here is a general strategy to strike that balance: start with an overfitting model function that tracks tiny fluctuations in the data, and progressively make it smoother until you hit a good middle ground. That idea of smoothing out the model function is called “regularization,” and is the subject of this section.

In the previous chapter, we took the first step of the process I just described: we created a deep neural network that overfits the data at hand. Let’s take a closer look at that network’s model, and afterwards we’ll see how ...