Machine Learning for OpenCV

Understanding the value of a validation set

Following our best practice of splitting the data into training and test sets, we might be tempted to tell people that we have found a model that performs with 97.4% accuracy on the dataset. However, our result might not necessarily generalize to new data. The argument is the same as earlier on in the book when we warranted the train-test split that we need an independent dataset for evaluation.

However, when we implemented grid search in the last section, we used the test set to evaluate the outcome of the grid search and update the hyperparameter k. This means we can no longer use the test set to evaluate the final data! Any model choices made based on the test set accuracy would leak information ...