Robust Python

Chapter 24. Mutation Testing

When weaving your safety net of static analysis and tests, how do you know that you are testing as much as you can? Testing absolutely everything is impossible; you need to be smart in what tests you write. Envision each test as a separate strand in your safety net: the more tests you have, the wider your net. However, this doesn’t inherently mean that your net is well-constructed. A safety net with fraying, brittle strands is worse than no safety net at all; it gives the illusion of safety and provides false confidence.

The goal is to strengthen your safety net so that it is not brittle. You need a way to make sure your tests will actually fail when there are bugs in your code. In this chapter, you will learn how to do just that with mutation testing. You’ll learn how to perform mutation testing with a Python tool called mutmut. You’ll use mutation testing to inspect the relation between your tests and code. Finally, you’ll learn about code coverage tools, how best to use those tools, and how to integrate mutmut with your coverage reports. Learning how to do mutation testing will give you a way to measure how effective your tests are.

What Is Mutation Testing?

Mutation testing is the act of making changes in your source code with the intent of introducing bugs.¹ Each change you make in this fashion is known as a mutant. You then run your test suite. If the tests fail, it’s good news; your tests were successful in eliminating the mutant. However, ...