Beautiful Testing

An AspectJ Example

In our experiments, we applied these mutation operators to medium-sized programs such as the AspectJ core, a 100,000-line Java package with a test suite of 321 unit tests, taking 21 seconds for a single run. We obtained 47,000 mutations, which, using the optimizations outlined previously, were evaluated in 14 CPU hours. This means that if you happen to have a single-core machine, you can run a mutation test overnight; if your machine has eight cores, you can run it during an extended lunch. (And if you’re running the latest and greatest, expect even lower times—or larger test and program sizes.)

The AspectJ test suite detected only 53% of the executed mutations. Some of the undetected mutations indicate real trouble. For instance, consider the staticAndInstanceConflict() method (Example 18-3) from the AjProblemReporter class in AspectJ. This method decides whether an error should be reported and eventually reports it to its superclass. However, if we mutate the method such that the call to super.staticAndInstanceConflict() is suppressed, the error reports are suppressed as well.

Example 18-3. A mutation that suppresses a method call (the AspectJ test suite does not detect this)

public void staticAndInstanceConflict(MethodBinding currentMethod, MethodBinding inheritedMethod) { if (currentMethod instanceof InterTypeMethodBinding) return; if (inheritedMethod instanceof InterTypeMethodBinding) return; super.staticAndInstanceConflict(currentMethod, inheritedMethod);  ...