Making Software

Reading Qualitative Research

Having described what qualitative methods are, we now turn to a discussion of how to read qualitative studies like the ones that appear throughout this book. For example, what does a particular study teach? When can you trust a study’s results? When can you generalize a study’s results to the larger world? To discuss these issues, let’s consider The Errors of TeX, published in 1989 by the Turing Award winner Donald Knuth [Knuth 1989].

In this classic article, Knuth analyzes more than 850 errors he logged while writing the TeX software. The study, as Knuth described it, was “to present a list of all the errors that were corrected in TeX while it was being developed, and to attempt to analyse those errors.” Knuth describes the rationale for his approach as overcoming the limitations of quantitative methods:

The concept of scale cannot easily be communicated by means of numerical data alone; I believe that a detailed list gives important insights that cannot be gained from statistical summaries.

What did Knuth discover in this study? He presents 15 categories of errors, gleaned from a much larger catalog, and then describes them with examples from his log. For example, Knuth describes the “blunder or blotch” category, which included program statements that were syntactically correct but semantically wrong. The root cause of these errors was variable names that were closely related conceptually but led to very different program semantics (e.g., reversing variables ...