Mastering Scala Machine Learning

Regularization

The regularization was originally developed to cope with ill-poised problems, where the problem was underconstrained—allowed multiple solutions given the data—or the data and the solution that contained too much noise (A.N. Tikhonov, A.S. Leonov, A.G. Yagola. Nonlinear Ill-Posed Problems, Chapman and Hall, London, Weinhe). Adding additional penalty function that skews a solution if it does not have a desired property, such as the smoothness in curve fitting or spectral analysis, usually solves the problem.

The choice of the penalty function is somewhat arbitrary, but it should reflect a desired skew in the solution. If the penalty function is differentiable, it can be incorporated into the gradient descent process; ridge regression ...