D.5. Holding your model back
During the model.fit(), the gradient descent is over-enthusiastic about pursuing the lowest possible error in your model. This can lead to overfitting, where your model does really well on the training set but poorly on new unseen examples (the test set). So you probably want to “hold back” on the reins of your model. Here are three ways to do that:
- Random dropout
- Batch normalization
In any machine learning model, overfitting will eventually come up. Luckily, several tools can combat it. The first is regularization, which is a penalization to the learned parameters at each training step. It’s usually, but not always, a factor of the parameters themselves. L1-norm and L2-norm ...