Making Software

Conclusion

In the previous section I discussed some SRs that offer new insights into software engineering. Many other SRs and mapping studies over the past few years have covered a variety of topics (see [Kitchenham et al. 2009b], [Kitchenham et al. 2010a], the IST virtual special issue,^[10] and Chapter 12). I believe these studies should start to change the way we do research in software engineering. Anyone interested in a particular topic needs to check for SRs addressing the topic. If no SRs exist, it may be a good idea to do one. If there are existing reviews, they can act as the starting point for your own literature search or can point you to topic areas where new research is necessary.

However, SRs have obvious limitations. First, just because something claims to be an SR doesn’t mean it was necessarily a high-quality review. You should read SRs as critically as you read any other research paper. You can use Greenhalgh’s evaluation criteria [Greenhalgh 2000] or the five criteria used by the Centre for Reviews and Dissemination [Centre for Reviews and Dissemination 2007]:

• Are the review’s inclusion and exclusion criteria described and appropriate?

• Is the literature search likely to have covered all relevant studies?

• Were the studies synthesized?

• Did the reviewers assess the quality/validity of the included studies?

• Were the basic data/studies adequately described?

The second major problem is that SRs rely on the availability of high-quality primary studies. The studies discussed ...