Site Reliability Engineering, 2nd Edition

Chapter 14. Testing

By Aman Shaikh, Elliot Williams, Forrest Seiwald, Joanna Wijntjes, Joon Lee, Niall Richard Murphy, Nicholas Crespi, Thomas C. Hubschman, and Yuri Grinshteyn with Alex Perry, Max Luebbe, and Carmela Quinito

In 1882, Benjamin Brewster wrote in a Yale student magazine: “In theory, theory and practice are the same; in practice, they are not.” This is as good a one-sentence summary of why testing exists as there is. It is also a clue about why testing is more important than ever in today’s era of massive distributed systems, complex production environments, and rapidly expanding deployment of AI-authored or AI-related software, none of which would be possible without it.

You’ve probably mostly encountered testing in the context of the software development lifecycle (SDLC), where it’s the phase before deployment and, as a result, before the new software gets put in front of users. This traditional role of testing is probably still the most common way it’s used today, though that’s changing fast. Conventionally, there’s some kind of procedure, sometimes manual, sometimes automatic, and often it’s a gating procedure: you have to pass the tests before software gets deployed. The tests are understood to provide some assurance that the software works, and there’s some evidence to suggest that the sooner you catch an error, the “cheaper” it is to fix, and the better for customers.¹ Similarly, it’s understood that having fixed the error, the tests pass, the software is ...