Machine Learning Algorithms - Second Edition

Lasso

A Lasso regressor imposes a penalty on the L₁ norm of w to determine a potentially higher number of null coefficients:

The sparsity is a consequence of the penalty term (the mathematical proof is non-trivial, but we have provided an intuitive explanation in Chapter 2, Important Elements in Machine Learning). A Lasso constraint yields a shrinkage too, but the dynamic is a little bit different from Ridge. When using an L₁ norm, the partial derivatives computed with respect to the coefficients can be only +1 or -1, according to the sign of θ_i. Hence, the constraint forces the smallest components to move towards zero with higher speed because ...