Experiment!: Planning, Implementing and Interpreting

8.5 The Power of a Test

As we have seen, it is possible that extreme values occur purely by chance. This means that there is a small probability that our significance test rejects the null hypothesis even when it is true. This is called a Type I error and the probability of making one is, of course, equal to our significance level, α. This may be illustrated using Figure 7.15, which represents the reference distribution connected with a null hypothesis. It is perfectly possible to obtain values in the shaded tails of this distribution also when the null hypothesis is true, but our significance test will reject the null hypothesis when values that occur with a probability of less than α = 0.05 are drawn.

When designing an experiment it is also important to be aware of a different risk: that of accepting the null hypothesis when it is, in fact, false. This is called a Type II error and the probability of making one is represented by the symbol β. Figure 8.4 illustrates how this risk occurs. The solid curve on the right is the reference distribution associated with the null hypothesis. It could, for example, be the t-distribution connected with Sven's fuel consumptions if the true mean were 6.0 liters per 100 km. Let us say that the null hypothesis is false. This means that we are sampling from another population, which is represented by the left, dashed curve. In actual situations we never know exactly which distribution we are sampling from, so it is only drawn here for illustration. ...