Validation methods
ML is a highly iterative process. Typically hundreds, thousands, and sometimes millions of variations are generated in the process. At this amount, even with a low probability of false positive results, you are essentially guaranteed to have not just one, but several false positives. The traditional application of statistics with a 95% confidence level breaks down at this amount of iterations. Most statistical tests assume that only one hypothesis is being tested. With ML, you are trying out thousands to millions of hypotheses in the search for the optimal model. You will certainly have models that pass all the statistical tests, just by sheer odds.
Following the principle of mistrust, no model is considered acceptable ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access