Common Errors in Statistics (and How to Avoid Them), 4th Edition

BEFORE YOU DRAW CONCLUSIONS

Insignificance

If the p-value you observe is greater than your predetermined significance level, this may mean any or all of the following:

1. You have measured the wrong thing, gone about measuring it the wrong way, or used an inappropriate test statistic.

2. Your sample size was too small to detect an effect.

3. The effect you are trying to detect is not statistically significant.

Practical Versus Statistical Significance

If the p-value you observe is less than your predetermined significance level, this does not necessarily mean the effect you have detected is of practical significance; see, for example, the section on measuring equivalence. For this reason, as we discuss in Chapter 8, it is essential that you follow up any significant result by computing a confidence interval, so readers can judge for themselves whether the effect you have detected is of practical significance.

And do not forget that at the α percent significance level, α-percent of your tests will be statistically significant by chance alone.

Missing Data

Before you draw conclusions, be sure you have accounted for all missing data, interviewed nonresponders, and determined whether the data were missing at random or were specific to one or more subgroups.

During the Second World War, a group was studying planes returning from bombing Germany. They drew a rough diagram showing where the bullet holes were and recommended that those areas be reinforced. A statistician, Abraham Wald ...