Hands-On Transfer Learning with Python

Weight-decay

It can be observed that an overfitted model—like the polynomial in the preceding example—has very large weights. To prevent this, a penalty term, Ω, can be added to the objective function, which will drive the weights closer to the origin. Thus, the penalty term should be a function of the norm of the weights. Also, the effect of the penalty term can be controlled by multiplying with a hyperparameter, α. So our objective function becomes: E(w) + αΩ(w). The popularly-used penalty terms are:

• L² regularization: Penalty term is given by . In regression literature, this is called ridge regression.
• L¹ regularization: Penalty term is ...