Beautiful Testing

Chapter 18. Seeding Bugs to Find Bugs: Beautiful Mutation Testing

Andreas Zeller

David Schuler

Suppose you are a test manager of a software project. You are at the end of a development cycle: all the new features are there, and all the tests have passed. Now is the time for you to decide: Are you ready for release?

In principle, everything should be settled, as all tests have passed. However, the fact that all your tests have passed may not mean much if your test suite is not good enough. If your test suite does not properly check the program outcome, for instance, your tests may pass even though the result is wrong. So, how can you check whether your test suite is effective in finding defects? In this chapter, we will explore a simple, elegant, and indeed beautiful way to test the quality of a test suite—namely by systematically seeding artificial defects into the program and checking whether the test suite finds them.

Assessing Test Suite Quality

Engineers and managers like to quantify things, and the quality of a test suite is no exception. To measure the quality of a test suite, various coverage metrics have been developed and deployed in practice. Best known is code coverage, where we check individual statements for whether they have been executed in a test suite. Obviously, if a test suite does not execute a statement, this calls for trouble. Suppose the statement has a bug that triggers a failure each time it is executed. If the test never reaches that statement, the bug will ...