Hands-On Machine Learning for Algorithmic Trading

Optimal stopping for backtests

In addition to limiting backtests to strategies that can be justified on theoretical grounds as opposed to as mere data-mining exercises, an important question is when to stop running additional tests.

Based on the solution to the secretary problem from optimal stopping theory, the recommendation is to decide according to the following rule of thumb: test a random sample of 1/e (roughly 37%) of reasonable strategies and record their performance. Then, continue tests until a strategy outperforms those tested before.

This rule applies to tests of several alternatives with the goal to choose a near-best as soon as possible while minimizing the risk of a false positive.