Common Errors in Statistics (and How to Avoid Them), 4th Edition

DRAWING CONCLUSIONS

Found data (nonrandom samples) can be very useful in suggesting models and hypotheses for further exploration, but without a randomized study, formal inferential statistical analyses are not supported [Greenland, 1990; Rothman, 1990]. The concepts of significance level, power, p-value, and confidence interval apply only to data that has arisen from carefully designed and executed experiments and surveys.

A vast literature has grown up around the unease researchers feel in placing too much reliance on p-values. Examples include Selvin [1957], Yoccoz [1991], Badrick and Flatman[1999], Feinstein [1998], Johnson [1999], Jones and Tukey [2000], McBride, Loftis, and Adkins [1993], Nester [1996], Parkhurst [2001], and Suter [1996].

The vast majority of such cautions are unnecessary providing we treat p-values as merely one part of the evidence to be used in decision making. They need to be viewed and interpreted in the light of all the surrounding evidence, past and present. No computer should be allowed to make decisions for you.

A failure to reject may result from any of the following:

1. A Type II error

2. Insensitive or inappropriate measurements

3. Additional variables being confounded with the variable of interest

4. Too small a sample size

This is another reason why the power of your tests should always be reported after correcting for missing data.

A difference that is statistically significant may be of no practical interest. Take a large enough sample ...