Machine Learning for OpenCV 4 - Second Edition

Assessing the significance of our results

Assume for a moment that we implemented the cross-validation procedure for two versions of our kNN classifier. The resulting test scores are—92.34% for Model A and 92.73% for Model B. How do we know which model is better?

Following our logic introduced here, we might argue for Model B because it has a better test score. But what if the two models are not significantly different? These could have two underlying causes, which are both a consequence of the randomness of our testing procedure:

• For all we know, Model B just got lucky. Perhaps we chose a really low k for our cross-validation procedure. Perhaps Model B ended up with a beneficial train-test split so that the model had no problem classifying ...