Making Software

The State of Evidence Today

Looking back, we now realize how naive our definitions of convincing evidence were. Elegant, statistically strong, replicated evidence turns out to be harder to find than we ever thought, and moreover, often doesn’t satisfy the more relevant goals we have for our research.

Challenges to the Elegance of Studies

We’ve found that elegant studies may be persuasive to those with sufficient training in research, but can be hard to explain to practitioners because they generally introduce constraints and simplifications that make the context less representative of real development environments. For example, the Basili and Selby study applied the techniques under study only to “toy” problems, none more than 400 lines of code, in an artificial environment. This study is often cited, and although it has been the subject of many replications, it does not seem that any of them used substantially larger or more representative applications [Runeson et al. 2006]. Although this elegant study has made a strong contribution to our understanding of the strengths and weaknesses of different approaches for removing defects from code, it is perhaps not ideal for a substantial part of our thinking on this subject to come from relatively small code segments.

Challenges to Statistical Strength

There is surprisingly little consensus about what constitutes “strong” statistics for real-world problems. First, there is the issue of external validity: whether the measures that are being ...