Common Errors in Statistics (and How to Avoid Them), 4th Edition

REVIEW THE SAMPLING DESIGN

Were the baseline values of the various treatment groups comparable? The baseline values (age and bone density) of the various groups studied by Fujita et al. [2000] were quite different, casting doubt on the reported findings.

Were the controls appropriate? The data quality assessment performed by Kelly et al. [1998] focused on the measured concentrations of various radioactive constituents in soil and creek sediments. At issue was whether radiation from the sediments exceeded background levels. Figure 4.1, taken from Figure 7 of their report, “shows that a comparison of background and site data for plutonium raises questions about the appropriateness of the plutonium background data, since the site data had lower levels than the background data.”

FIGURE 4.1. In these box and whisker plots taken from Kelly et al. (1998), U.S. Dept Energy, the site data had lower levels of plutonium than the background data.

Was the blinding effective? A subsample of those completing printed and on-line surveys should be contacted for personal interviews to verify their responses. (See, for example, Nunes, Pretzlik, and Ilicak, 2005).

Similarly, a subsample of nonresponders should also be contacted.

Berger [2005] is skeptical about contacting investigators: “If there is selection bias, and we ask those investigators who caused it which treatments they think each patient ...