Test Driven Machine Learning

Testing with simulation

A recurring theme in this book is to use what's known as Monte Carlo methods to quantify the quality of our algorithm. Using this technique, we will time and again create groups of random data to see how well our algorithms (ignorant of how we created the data) perform at detecting the differences. In the previous chapter, most of the tests that we developed had deterministic test cases. In this chapter, the test cases will be specified by the parameters that we use, but all of the data that we create will be random.

First things first, we need to develop a small framework that we can use to run our simulations. In order to do this, we should more clearly articulate our scenario. Here's a more thorough definition.

Imagine ...