Statistical Rethinking

6    Overfitting, Regularization, and Information Criteria

Nicolaus Copernicus (1473–1543): Polish astrologer, ecclesiastical lawyer, and blasphemer. Famous for his heliocentric model of the solar system, Copernicus argued for replacing the geocentric model, because the heliocentric model was more “harmonious.” This position eventually lead (decades later) to Galileo's famous disharmony with, and trial by, the Church.

This story has become a fable of science's triumph over ideology and superstition. But Copernicus' justification looks poor to us now, ideology aside. There are two problems: The model was neither particularly harmonious nor more accurate than the geocentric model. The Copernican model was very complicated. In fact, it had similar ...