Trading Systems and Methods, + Website, 5th Edition

CHAPTER 21

System Testing

Numbers are like people; torture them enough and they will tell you anything.

—Anonymous (quoted by John Ehlers in Rocket Science for Traders)

System testing remains a fight between computer power and common sense. Before 1980, testing a trading system was a simple concept. Conscientious traders and analysts applied their ideas to charts, marking the buy and sell signals manually and keeping a hand-written record of the trades. The process took a long time; therefore, they were careful not to begin unless they were reasonably sure that the method had a good chance of success. Ideas came from an understanding of the fundamentals or the awareness of market patterns, plus valuable experience. The tabulation of profitable results was verification that the idea was sound.

By the early 1990s, you were at the high end of technology if you had daily data that could be read by a computer and a program such as TradeStation or MetaStock to give you charts and indicators that could be combined into a trading system. Best of all, these programs showed the historic profitability of the trading strategies and, in particular, the best strategy. Moving averages and the RSI were still sophisticated tools, but the ability to test for the best calculation period and combine indicators began a step down a path that has now become very complex and taken the industry further from the simple solution.

That process has continued to advance, but not always for the better. Those ...