The R Book, 2nd Edition

Chapter 21

Meta-Analysis

There is a compelling case to be made that analysts should look at the whole body of evidence, rather than trying to understand individual studies in isolation. The systematic review of a body of evidence is known as meta-analysis. The idea is to draw together all of the appropriate studies that have addressed the same question, and calculate an overall effect and an overall measure of uncertainly for that effect. Of course, there are plenty of issues with this, among them the following:

• What do we mean by ‘the same question’?
• What is an appropriate study?
• What makes a study inappropriate ?
• How different can a study be, and still be worth including in the meta-analysis?
• What are the publication biases in the various studies?
• Are the effects fixed or random?

The central idea is to remove as much as possible of the subjectivity that was such a feature of old-fashioned narrative reviews. In an ideal world, we should be able to extract from every published study the exact question addressed, the effect size, the variance of that effect, the replication, and enough detail on the methods used to be confident that the study was comparable with the others that we have already included. Then we simply calculate a weighted average effect size and an appropriate weighted average measure of the unreliability of that overall effect. And that would be it.

If the effect is consistent across experiments, this gives considerable confidence in its generality, and the ...