O'Reilly logo

Swing Extreme Testing by Tim Lavers, Lindsay Peters

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 20. Flaky Tests

Tests that fail intermittently are the biggest obstacle to an automated testing process. As we deal with them all the time, we even have a nickname for them. We call them 'flaky' tests.

The problem with having flaky tests is that they remove the element of reproducibility from our testing process. If our tests pass, we don't know whether all the bugs we're testing against have actually been fixed, or whether they just happened not to manifest themselves in that testing run. If our tests fail, this could be due to either production errors or testing errors, or both.

If we can't be sure of our test results, we can't be sure of our software.

Furthermore, even if we are fairly confident that the "flakes" are due to testing errors ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required