Making Software

What Should You Believe About These Results?

Designing empirical studies is just like designing software systems: it is impossible to create a bug-free system, and we often make design mistakes and create systems with flaws and weaknesses. The main question in both cases is: do the problems negate the usefulness of the software systems or the empirical studies?

There are three main questions we need to address in order to determine how good our study is and whether you can justifiably use our results: 1) are we measuring the right things, 2) are there other things that might be the explanations for what we see (i.e., did we do it right?), and 3) what do our results apply to (i.e., what can we do with the results)?

Are We Measuring the Right Things?

We believe that we have a very strong argument to support our claim that we have addressed the critical issues in understanding software development faults and their implications. We address the fundamental issues in fault studies: the faults that occur, how hard it is to find and fix them, their underlying causes, and how might we prevent them, detect them, or ameliorate them. In addition, we addressed a question raised in response to the interface fault studies we had done earlier: which are harder to find and/or fix, interface or implementation faults? Strong support for this comes from the consistency and mutual support provided by the Chi-Square analysis, in which there are very strong relationships between the faults detected, their ...