Trading Systems and Methods, 6th Edition

CHAPTER 21System Testing

Numbers are like people; torture them enough and they will tell you anything.

– Anonymous (quoted by John Ehlers in Rocket Science for Traders)

System testing remains a fight between computer power and common sense. Before computers, testing a trading system was a simple concept. Traders and analysts applied their ideas to charts, marking the buy and sell signals manually and keeping a hand-written record of the trades. The process took a long time, so it wasn't started unless the idea was thought to be good. Those ideas came from an understanding of the fundamentals, awareness of market patterns, and valuable experience. A profitable result verified the concept.

By the early 1990s, you were at the high end of technology if you had a computer with daily data updates, and a program such as TradeStation or MetaStock to give you charts and indicators that could be combined into a trading system. These programs could show the historic profitability of the trading strategies and, even better, the best strategy. Moving averages and the RSI were still sophisticated tools, but the ability to test for the best calculation period and combine indicators began a step down a path that has now become very complex and has taken the industry farther from the simple solutions.

That process has continued to advance, but not always for the better. Those who have applied diligence, selectivity, and most of all common sense to automating systems are likely to have gotten ...