Making Software

Threats to Validity

The primary construct in our study is the use of bug reports as the logical unit of change to the analyzed systems. The first threat to the validity of this construct comes from the simple pattern-matching technique we use to associate commits with bug reports based on their description field. This technique does not guarantee that all of the links between bug reports and commits are found; some commit descriptions may report the associated bug ID in a nonstandard format that is missed by our regular expression, whereas other commits may describe the bug they address without explicitly mentioning the bug ID. In particular, we were only able to match approximately 20% of the commits for Evolution, which restricted our analysis to a small subset of the system’s full history.

A second threat to the construct validity is due to noise in the Bugzilla repositories. We assumed that each bug report corresponds to a single logical change to the system, but this does not hold for all bug reports. For example, some reports describe routine software maintenance (e.g., upgrading a third-party library that is used by the system to the latest version) that could have very different characteristics than the defect reports and enhancement requests that we intended to study.

The internal validity of our study concerns the relationship between our data and the conclusions reached. Many of the conclusions are subjective interpretations of the data we present, and other inferences ...