Making Software

Chapter 3. What We Can Learn from Systematic Reviews

Barbara Kitchenham

As a strong advocate of evidence-based software engineering ([Dybå et al. 2005], [Kitchenham et al. 2004]), I am also a strong advocate of systematic reviews (SRs) ([Kitchenham 2004], [Kitchenham and Charters 2007]). These are sometimes referred to as systematic literature reviews in software engineering to avoid confusions with inspection methods (i.e., methods for reading and reviewing software engineering documents or code). We cannot have evidence-based software engineering without a sound methodology for aggregating evidence from different empirical studies. SRs provide that methodology.

SRs have been in widespread use in other disciplines for decades. Each SR is launched by a researcher to investigate all available evidence that supports or refutes a particular “topic of interest,” which in software engineering typically involves asking about the effect of a method or process. A researcher conducting an SR selects empirical studies that are relevant to the particular research question, assesses the validity of each one, and then determines the trend shown by those studies. Thus, SRs aim to find, assess, and aggregate all relevant evidence about some topic of interest in a fair, repeatable, and auditable manner.

This chapter introduces the value of SRs to readers with a general interest in empirical software engineering. I also aim to help novice researchers (such as PhD students)—who might be looking for ...