Machine Learning With Go

Holdout set

We are making progress to ensure that our models generalize using training and test sets. However, imagine the following scenario:

1. We develop a first version of our model based on our training set.
2. We test this first version of our model on the test set.
3. We aren't satisfied with the result on the test set, so we loop back to step 1 and repeat.

This process might seem logical, but you are probably already seeing a problem that can result from this procedure. We can actually overfit our model on the test data by iteratively exposing the model to our test set.

There are a couple of ways to deal with this extra level of overfitting. The first is by simply creating another split of our data called a holdout set (also known as a ...