Experiment!: Planning, Implementing and Interpreting

9.1 Statistics and the Scientific Method

We saw in Chapter 6 that many experiments do not involve statistical tests at all. The interesting effect may be so strong that it is not necessary to use statistics to separate it from the noise. In many cases, however, statistical tools are indispensable components of the experimental strategy, especially when effects are weak or when background factors are important. The combination of statistical and experimental method is called design of experiments. Fisher [1] originally used the term for planning an experiment with a particular significance test in mind. The teatime experiment in the last chapter is a good example of this, borrowed from his book The Design of Experiments. It shows how an experiment can be designed to discern the potential weak effect of a single factor by applying replication and randomization. Today, as we shall see, the term “design of experiments” has been extended to include orthogonal multifactor experiments where the analysis is often based on linear models of the data rather than pure significance tests.

When significance tests are used, it is important not to confuse the statistical hypothesis with the scientific one. Scientific hypotheses go beyond particular sets of data and make general statements about the world. As explained in Chapter 6, they may even be a basis for explanatory theory if they involve a mechanism. Statistical hypotheses are not explanatory; they only make simple statements about mathematical ...