The Evaluation and Optimization of Trading Strategies, 2nd Edition

CHAPTER 13

The Many Faces of Overfitting

Over the years, a number of different expressions have been used somewhat interchangeably to describe problems that arise in the process of evaluating and optimizing a trading strategy. These terms include overoptimization, overfitting, curve-fitting, and data mining.

These words are also typically used in a pejorative and dismissive manner. There are still those who derisively, and with great ignorance, dismiss all optimization as curve-fitting, which means for those in that camp that an optimized strategy is, almost by definition, a fantasy or a delusion.

To entirely discard as garbage the very idea of optimization because an improperly optimized strategy has failed in real-time trading is itself, however, a product of ignorance and poor logic. It is the same type of logic that would cause someone who has seen only blond women to overgeneralize and conclude that all women are blondes.

It is not a coincidence, of course, that those who conclude that optimization is useless are those who have probably never seen a correctly optimized trading strategy.

It is incorrect to use these four terms interchangeably. Overoptimization and overfitting are identical in meaning. This equivalence will be seen clearly once the term “overfitting” is precisely defined.

Curve-fitting and data mining are often taken to have a similar sense and this too is understandable. They are two different processes, though, with little, in fact, relationship to one another. ...