Programming Machine Learning

The Threat of Overfitting

Since Part I of this book, we’ve been using two separate sets of examples—one for training our algorithms, and one for testing them. Let’s refresh our memory: why don’t we use the same examples for both training and testing?

I’ll use a metaphor to answer that question. Imagine this: you’re teaching basic math to a class of young kids. You already prepared 60 multiple-answer multiplication quizzes. You plan to assign most of those quizzes as homework. You also plan to select 10 quizzes for a final test, to check how well the kids are learning.

To recap, you want to split the quizzes in two groups: “homework” and “test.” Here are two options: either you assign all 60 quizzes to “homework,” and then re-use 10 of them in ...