Experiment!: Planning, Implementing and Interpreting

12.1 Turning Data into Information

It is sometimes said that a good experiment analyses itself. It is true that careful planning and data collection make the analysis easier and this is why we devoted so much space to them in the two preceding chapters. But it is equally true that we would not make experiments if we knew the outcome. Effects may turn out to be very weak or there may be unexpected aspects of the data that make them difficult to analyze. And even when the data are good, it may not be straightforward to present the results clearly and concisely.

Even though the preparations in the preceding two chapters will improve our prospects, they do not guarantee success. They say that the road to hell is paved with good intentions. You will know what this means if you spend months planning an experiment, getting your setup to work and collecting data, only to find that the data are insufficient to support a clear conclusion. One way to avoid unpleasant surprises when evaluating the data is to do parts of the analysis already during the data collection phase. This allows us to anticipate potential problems. If the results are not promising, parts of the experiment may have to be redesigned. This is, of course, also the reason for the recommendation in the last chapter that we should not collect all the data in one go.

In the last chapter we mentioned that it is sometimes necessary to revisit the planning phase during the data collection phase. In the same way, we may occasionally ...