Hands-On Machine Learning for Algorithmic Trading

Data-snooping and backtest-overfitting

The most prominent challenge to backtest validity, including to published results, relates to the discovery of spurious patterns due to multiple testing during the strategy-selection process. Selecting a strategy after testing different candidates on the same data will likely bias the choice because a positive outcome is more likely to be due to the stochastic nature of the performance measure itself. In other words, the strategy is overly tailored, or overfit, to the data at hand and produces deceptively positive results.

Hence, backtest performance is not informative unless the number of trials is reported to allow for an assessment of the risk of selection bias. This is rarely the case in practical ...