Mechanical Trading Systems: Pairing Trader Psychology with Technical Analysis

SYSTEM DEVELOPMENT PROCESS

Out-of-Sample Study

The out-of-sample or walk-forward study is probably one of the most important aspects of the system development process. This procedure of setting aside a statistically significant (and most up-to-date¹⁵) portion of the data series to ensure that the system is behaving as forecasted is crucial to system developers because it enables us to test the system prior to committing actual funds.

The most essential aspect of the out-of-sample data window is its integrity. Data integrity is defined here as the inability of our in-sample results to bleed through into our out-of-sample data. Although data integrity of the out-of-sample window might appear to be a given prerequisite, it never hurts to restate the obvious, especially since failure to adhere to this rule will necessarily compromise the value of all out-of-sample testing. Other, less critical characteristics of the out-of-ample window are that it generally should contain somewhere between 10 to 20 percent of the data displayed within the in-sample window.¹⁶

Although out-of-sample results never look exactly like those of our in-sample performance, there should be a strong positive correlation between the two data sets. If walk-forward performance yields results that are drastically different from those of in-sample data (e.g., postoptimization drawdowns exceeding 15 percent of in-sample), we probably should abandon the trading system. This seemingly drastic response to excessive drawdowns ...