The Art of Machine Learning

3BIAS, VARIANCE, OVERFITTING, AND CROSS-VALIDATION

We now look in detail at a vital topic touched on in Sections 1.7, 1.12.4, and 2.2.5—overfitting. In this chapter, we’ll explain what bias and variance really mean in ML contexts and how they affect overfitting. We’ll then cover a popular approach to avoiding overfitting known as cross-validation.

The problem of overfitting exemplifies the point made in the title of this book: ML is an art, not a science. There is no formulaic solution to various problems, especially overfitting. Professor Yaser Abu-Mostafa of Caltech, a prominent ML figure, once summed it up: “The ability to avoid overfitting ...