Common Errors in Statistics (and How to Avoid Them), 4th Edition

NONRANDOM SAMPLES

Quite often, particularly when exploring the implications of proposed government policies, we are forced to make do with found (or observed) data; that is, we access data that do not result from planned or controlled experiments. We consider the potential sources of error to be found in epidemiological studies and in case-control studies.

Epidemiology

It is common in epidemiological investigations to compare the events that take place in a specific location before and after a specific policy is implemented and/or to compare the events that take place in a specific time period in two distinct locations, one where the policy is implemented and one where it is not.

Marshall et al. [2011] examined the population-based overdose mortality rates for the period before (Jan 1, 2001, to Sept 20, 2003) and after (Sept 21, 2003, to Dec 31, 2005) the opening of the Vancouver Safe-Injection Facility. They reported a practical as well as statistically significant decrease in the immediate (500 meter) area in contrast to a minor decrease in the fatal overdose rate in the rest of the city.

A rebuttal by Pike et al. [2011]¹ noted the following sources of error in the Marshall report:

• The choice of control period was suspect; 2001 was a year of markedly higher heroin availability and overdose fatalities than all subsequent years.
• Confounding variables were neglected; other changes in government policy may have affected the results. For example, 50–66 extra police were specifically ...