Common Errors in Statistics (and How to Avoid Them), 4th Edition

PERMUTATION TESTS

First introduced by Pitman [1937, 1938], permutation tests are often lauded erroneously in the literature as assumption-free panaceas. Nothing could be further from the truth.

Permutation tests only yield exact significance levels if the labels on the observations are weakly exchangeable under the null hypothesis.³⁶ After eliminating the main effects in a multiway analysis of variance, the residuals are correlated, the correlation depending on the subscripts; they are not exchangeable. Thus the permutation test for interaction proposed by Still and White (1981) is not exact. Nor is the far more popular Kruskall–Wallace test. For the same reason, permutation tests cannot be successfully applied to the coefficients in a multivariate regression, though many have made the attempt and failed (see, for example, Oja, 1981; Kennedy, 1995).

On the other hand, permutation tests are the method of choice for the following:

• Two-sample multivariate comparisons
• Comparison of variances
• Crossover designs
• k-sample comparisons
• Type I censoring
• Contingency tables, whenever there are 12 or fewer observations in each subsample

Moreover, permutation methods can be used both to test hypotheses and to obtain interval estimates of parameters.

In other practical situations, such as the two-sample comparison of means (crossover designs being the exception) and bivariate correlation, permutation tests offer no advantage over parametric methods such as Student’s t and Pearson correlation. ...