D.5. Holding your model back

During the model.fit(), the gradient descent is over-enthusiastic about pursuing the lowest possible error in your model. This can lead to overfitting, where your model does really well on the training set but poorly on new unseen examples (the test set). So you probably want to “hold back” on the reins of your model. Here are three ways to do that:

  • Regularization
  • Random dropout
  • Batch normalization

D.5.1. Regularization

In any machine learning model, overfitting will eventually come up. Luckily, several tools can combat it. The first is regularization, which is a penalization to the learned parameters at each training step. It’s usually, but not always, a factor of the parameters themselves. L1-norm and L2-norm ...

Get Natural Language Processing in Action now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.