Programming Machine Learning

A Testing Conundrum

To see where the testing hurdle is, consider that we’re going to tune our neural network with an iterative process. That process is going to work like this:

1. Tune the network’s hyperparameters.
2. Train the network on the training set.
3. Test the network on the test set.
4. Repeat until we’re happy with the network’s accuracy.

This process is pretty much the ML equivalent of software development, so we can simply call it like that: “the development cycle.”

We already went through a few iterations of development in the previous chapter, when we measured the network’s performance with different batch sizes. However, we overlooked a distressing fact: the development cycle violates the Blind Test Rule. Here is why.

During development, we ...