Scala for Machine Learning - Second Edition

Regularization

The ordinary least squares method for finding the regression parameters is a specific case of the maximum likelihood. Therefore, regression models are subject to the same challenge in terms of overfitting as any other discriminative model. You are already aware that regularization is used to reduce model complexity and avoid overfitting as stated in Overfitting section of Chapter 2, Data Pipelines.

L_n roughness penalty

Regularization consists of adding a penalty function J(w) to the loss function (or RSS in the case of a regressive classifier) to prevent the model parameters (also known as weights) from reaching high values. A model that fits a training set very well tends to have many features variable with relatively large weights. ...