Beautiful Testing

Focusing on Impact

In our experiments, we were surprised to find thousands of undetected mutations, out of which an intriguing proportion were equivalent. We wanted to focus on the most valuable mutations, where a “valuable” mutation would be the one that helps improve the test suite most. (Of course, an equivalent mutant would be the least valuable.)

Remember what we said earlier about the risk distribution? We assumed that if some mutation in a component C could impact several other components, then the component C would be risky, and the more components impacted, the higher the risk. We want to reduce risk in our test suite. Therefore, we would focus on those undetected mutations that have the highest impact on other components. If I can find a mutation that changes the behavior all across my program, but my test suite does not care, I should have a good reason to improve it.

The question was about how to measure the impact of a mutation. So far, we have explored two ways of measuring impact:

Impact on code coverage

If a mutation leads to different execution paths being taken, it has a higher likelihood of changing the program behavior. We therefore count the number of methods in which the coverage changes after a mutation. The more methods, the higher the impact (Grün et al. 2009).

Impact on pre- and postconditions

Every method has a number of conditions that hold for its arguments and its return values. We automatically learn such conditions from executions, and then check whether ...