97 Things Every Java Programmer Should Know

Chapter 87. The Unreasonable Effectiveness of Fuzz Testing

Nat Pryce

Whether using test-driven development or not, programmers writing automated tests suffer from positive test bias:^{1, 2} they are more likely to test that the software behaves correctly when given valid input than that the software is robust when given invalid input. As a result, our test suites fail to detect entire classes of defects. Fuzz testing³ is an unreasonably effective technique for negative testing that is easy to include in existing automated test suites. Including fuzz tests in your test-driven development process will help you build more robust systems.

For example, we were extending the software of a widely used consumer product to fetch data from web services. Although we were careful to write robust networking code and test-drove negative as well as positive cases, fuzzing immediately uncovered a surprising number of inputs that would make the software throw unexpected exceptions. Many of the standard Java APIs that parse data throw unchecked exceptions, so the type checker hadn’t been able to ensure that the application handled all possible parsing errors. These unexpected exceptions could leave the device in an unknown state. In a consumer device, even one that can be updated remotely, that can mean an expensive increase in customer support calls or engineer callouts.

A fuzz test generates many ...