CHAPTER 9
Case Study Results and the Future of TA
PRESENTATION OF RESULTS
The primary objective of the case study was to demonstrate the application of two statistical inference methods suitable for the evaluation of rules discovered by data mining. As explained in Chapter 6, traditional significance tests are not suitable because they do not take into account the biasing effect of data mining. The two methods used were White’s reality check (WRC) and the Monte Carlo permutation (MCP).
A secondary objective of the case study was the possible discovery of rules with statistically significant returns when applied to the S&P 500 Index. Toward this end, a set of 6,402 rules described in Chapter 8 were back tested and evaluated.
With respect to the primary objective, the case study resoundingly demonstrated the importance of using significance tests designed to cope with data-mining bias. With respect to the second objective, no rules with statistically significant returns were found. Specifically, none of the 6,402 rules had a back-tested mean return that was high enough to warrant a rejection of the null hypothesis, at a significance level of 0.05. In other words, the evidence was insufficient to reject a presumption that none of the rules had predictive power.
The rule with the best performance, E-12-28-10-30,1 generated a mean annualized return of 10.25 percent, on detrended market data. In Figure 9.1, the rule’s return is compared to the sampling distribution produced by WRC. ...
