Making Software

Code Coverage

Code coverage is an important metric by which the extent of testing is often quantified. The main assumption behind coverage is that if a branch or a statement contains a flaw, it cannot be detected unless a test at least executes that statement or branch (and obtains an output contrary to the normal expectation). It can be argued that higher coverage should lead to detection of more flaws in the code and, if they are fixed, to better release quality. Although this assumption is widely believed, there is little evidence to show that higher code coverage results in fewer failures.

There are a number of potential flaws with the argument that higher code coverage leads to better quality. First, the coverage measure reflects the percent of statements covered but does not consider whether these statements are likely to contain a flaw. Therefore, it is possible to create two test sets with the same coverage measure but a markedly distinct ability to detect post-release defects. Second, the fact that a statement or a branch was executed does not imply that all possible data values have been exercised by the test. The customer/usage scenario could be totally different from what the tester perceives it to be. This is of particular concern for systems with simple control flow that can be easily covered with a few test cases. In our study on code coverage, based on qualitative interviews with engineers, the reasons for observing such a trend was attributed to several factors ...