Building Machine Learning Systems with Python - Third Edition

L1 and L2 penalties

We now explore these ideas in detail. Readers who do not care about some of the mathematical aspects should feel free to skip directly to the next section on how to use regularized regression in scikit-learn.

The problem, in general, is that we are given a matrix X of training data (rows are observations, and each column is a different feature), and a vector y of output values. The goal is to obtain a vector of weights, which we will call b*. The ordinary least squares regression is given by the following formula:

That is, we find vector b, which minimizes the squared distance to the target y. In these equations, we ignore ...