Making Software

Change We Can Believe In

To date, we have not realized our dream of evidence that is elegant, statistically sound, and replicable. And where we have found evidence in each of these categories, it has not always had the impact we hoped for. Perhaps we need to revisit our definitions of “convincing evidence.” Given the feedback in the previous section, is there a more practical definition of convincing evidence that should be motivating researchers?

A more feasible (if humble) definition is this: convincing evidence motivates change. We suspect that many of the authors in this book began their hunt for convincing evidence when they saw firsthand the problems and difficulties in real-world software development. The Holy Grail for such researchers tends to be the research results that can create real-world improvement.

To motivate change, some influential audience has to trust the evidence. One way to deal with the lack of rigor in experience reports or case studies might be to assign a “confidence rating” to each piece of evidence. The rating would attempt to reflect where each report fit on a spectrum that ranges from anecdotal or problematic evidence to very trustworthy evidence. Such evaluations are an essential part of the process proposed by Kitchenham for systematic reviews and aggregations of software engineering studies [Kitchenham 2004]. A simple scale aimed at helping to communicate confidence levels to practitioners can be found in a paper by Feldmann [Feldmann et al. 2006] ...