Regenerative Software

Chapter 5. Designing Evaluations

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 5th chapter of the final book.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

There is a problem at the center of software testing so basic that it has a name, and most working engineers have never heard it. To test a program, you run it on some input and check whether the output is correct. The second half of that sentence hides the hard part. Check it against what? For a sort routine, you can write down the right answer. For a fraud model, a pricing engine, a recommendation system, or almost anything else worth building, there is no independent authority sitting nearby that already knows the correct output. The field has called this the test oracle problem since the 1970s, and the point of the name is to insist on something engineers forget: the difficulty is not running the test, it is knowing what correct means.

For most of software history you could paper over the oracle problem, because code changed slowly and a human who understood it stayed close by. Regeneration removes both cushions. If a component is going to be deleted and rebuilt on demand, possibly by a machine, possibly many times ...