Taking Testing Seriously

CHAPTER 7How to Approach AI and Testing

—By James Bach and Michael Bolton

What If We Could Test by Magic?

Artificial intelligence is now a major factor to consider in all forms of knowledge work. Developments in AI keep coming. There's no way a book like this—constructed from the organic experiences and thoughts of humans from a certain era—can say exactly what is coming. Nevertheless, there is a helpful way to talk about artificial intelligence that is proof against all possible futures: we can talk about magic.

Consider a magic testing box. You don't know how it works, but—according to the leprechaun who gave it to you—when you place it near a product to be tested and perform the appropriate ceremonies of activation, it finds and reports bugs in that product. Pretty nice, huh? Not necessarily. To a professional tester—to anyone taking testing seriously—it may be more of a burden than a boon. This is because the magic box sets for you a puzzle that you cannot ignore: did it do a good job? Sure, the honest-looking leprechaun assured you that it “works.” But let's say that taking other people's word for what constitutes good testing is not your style.

Let's say you try it out. If it doesn't find bugs, was that because there were no bugs to find? Or did you forget to turn it on? Perhaps you did not make the correct offerings of incense and chocolate, and it operated at only half-strength? Perhaps it found bugs, but didn't find the good ones. We are tempted to write a very long ...