Common Errors in Statistics (and How to Avoid Them), 4th Edition

P-VALUES

The p-value is not the probability that the null hypothesis is true.

—Yoccoz [1991]

Before interpreting and commenting on p-values, it is well to remember that in contrast to the significance level, the p-value is a random variable that varies from sample to sample. There may be highly significant differences between two populations and yet the samples taken from those populations and the resulting p-value may not reveal that difference. Consequently, it is not appropriate for us to compare the p-values from two distinct experiments, or from tests on two variables measured in the same experiment, and declare that one is more significant than the other.

If we agree in advance of examining the data that we will reject the hypothesis if the p-value is less than 5%, then our significance level is 5%. Whether our p-value proves to be 4.9% or 1% or 0.001%, we will come to the same conclusion. One set of results is not more significant than another; it is only that the difference we uncovered was measurably more extreme in one set of samples than in another.

Note that, after examining the data, it is unethical to alter the significance level or to reinterpret a two-tailed test as if one had intended it to be one-tailed.

p-values need not reflect the strength of a relationship. Duggan and Dean [1968] reviewed 45 articles that had appeared in sociology journals between 1955 and 1965 in which the chi-square statistic and distribution had been employed in the analysis of 3 × 3 contingency ...