Common Errors in Statistics (and How to Avoid Them), 4th Edition

MODERN STATISTICAL METHODS

The greatest error associated with the use of statistical procedures is to make the assumption that one single statistical methodology can suffice for all applications.

From time to time, a new statistical procedure will be introduced or an old one revived along with the assertion that at last the definitive solution has been found. Parallel with the establishment of new religions, at first the new methodology is reviled, even persecuted, until, growing in the number of its adherents, it can begin to attack and persecute the adherents of other more established dogmas in its turn.

During the preparation of this text, an editor of a statistics journal rejected an article of one of the authors on the sole grounds that it made use of permutation methods.

“I’m amazed that anybody is still doing permutation tests … ” wrote the anonymous reviewer, “There is probably nothing wrong technically with the paper, but I personally would reject it on grounds of irrelevance to current best statistical practice.” To which the editor sought fit to add, “The reviewer is interested in estimation of interaction or main effects in the more general semi-parametric models currently studied in the literature. It is well known that permutation tests preserve the significance level but that is all they do is answer yes or no.”²

But one methodology can never be better than another, nor can estimation replace hypothesis testing or visa versa. Every methodology has a proper domain ...