Beautiful Testing

Watching the Watchmen

A common way to test the quality of quality assurance is to simulate a situation in which quality assurance should trigger an alarm. For instance, to test a watchdog, one could check whether the dog actually gives an alarm when facing a (staged) intruder. To test the quality assurance of a clothes company, one could insert a faulty cloth into the production chain and check whether quality assurance finds it. Secure systems are routinely tested using penetration testing, simulating an attack from a malicious source.

In 1971, Richard Lipton adapted this simulation concept to testing. His idea, written down in a paper called “Fault diagnosis of computer programs,” was to seed artificial defects, called mutations, into the software under test, and to check whether the test suite would find them. Such an artificial defect could be to change a constant, to remove a function call, or to negate a branching condition. If the test suite fails to detect the mutation, it would likely miss real defects, too, and thus must be improved.

Looking back to Example 18-1, we could mutate the implementation of the BuffereredWriter.write() method such that it does nothing at all—for instance, by inserting a return statement at the beginning of the method. The test in Example 18-1, however, would not detect this mutation, indicating that it would miss real defects, too. (And indeed it would, as we already discussed its inadequacy.) Example 18-2 shows an improved version of the test. ...