Machine Learning Algorithms - Second Edition

Ridge

Ridge regression (also known as Tikhonov regularization) imposes an additional shrinkage penalty to the ordinary least squares cost function to limit its squared L₂ norm:

X is a matrix containing all samples as rows and the term θ represents the weight vector. The additional term (through the alpha coefficient—if large it implies a stronger regularization and smaller values) forces the loss function to disallow an infinite growth of w, which can be caused by multicollinearity or ill-conditioning.

In the following diagram, there's a representation of what happens when a Ridge penalty is applied:

Bidimensional solution using a Ridge penalty ...