Programming Machine Learning

What You Just Learned

So far, we split our examples in a training set and a test set. This approach, however, tends to break down once we start tuning our system’s hyperparameters, because it sneakily leads us to optimize the system for the specific examples in the test set.

In this chapter, we switched to a more sophisticated approach, splitting the test set in two: a smaller test set, and a brand-new validation set. We’ll use the validation set for development, and the test set only once, for our final benchmark. During the final benchmark, the network comes in contact with the test set for the first time, so it should give us a reliable idea of the network’s accuracy on future production data.

One word of warning before we move on: this three-sets ...