Machine Learning for OpenCV

Assessing the significance of our results

Assume for a moment that we implemented the cross-validation procedure for two versions of our k-NN classifier. The resulting test scores are-- 92.34% for Model A, and 92.73% for Model B. How do we know which model is better?

Following our logic introduced here, we might argue for Model B because it has a better test score. But what if the two models are not significantly different? These could have two underlying causes, which are both a consequence of the randomness of our testing procedure:

• For all we know, Model B just got lucky. Perhaps we chose a really low k for our cross-validation procedure. Perhaps Model B ended up with a beneficial train-test split so that the model had no problem classifying ...