The Art of Machine Learning

9CUTTING THINGS DOWN TO SIZE: REGULARIZATION

A number of modern statistical methods “shrink” their classical counterparts. This is true for ML methods as well. In particular, the principle may be applied in:

• Boosting (covered in Section 6.3.8)
• Linear models
• Support vector machines
• Neural networks

In this chapter, we’ll see why that may be advantageous and apply it to the linear model case. This will also lay the foundation for material in future chapters on support vector machines and neural networks.

9.1 Motivation

Suppose we have sample data on human height, weight, and age. We denote the population means of these quantities by μ_ht, μ_wt and ...