Common Errors in Statistics (and How to Avoid Them), 4th Edition

Chapter 6

Testing Hypotheses: Choosing a Test Statistic

Forget “large-sample” methods. In the real world of experiments, samples are so nearly always “small” that it is not worth making any distinction, and small-sample methods are no harder to apply.

—George Dyke [1997].

Statistical tests should be chosen before the data are analyzed, and the choice should be based on the study design and distribution of the data, not the results.

—Cara H. Olsen

LIFE CONSTANTLY FORCES US TO MAKE DECISIONS. IF life were not so uncertain, the “correct” choice always would be obvious. But life is not certain and the choice is not obvious. As always, proper application of statistical methods can help us to cope with uncertainty, but cannot eliminate it.

In the preceding chapter on estimation, our decision consisted of picking one value or one interval out of an unlimited number of possibilities. Each decision had associated with it a potential loss, an amount that increased as the difference between the correct decision and our decision increased.

In this chapter on hypothesis testing, our choices reduce to three possibilities:

1. To embrace or accept a primary hypothesis.

2. To reject the primary hypothesis and embrace or accept one or more alternative hypotheses.

3. To forgo making a decision until we have gathered more data.

Among the most common errors in (prematurely) published work is the failure to recognize that the last decision listed above is the correct one.