97 Things Every SRE Should Know

Chapter 91. The Importance of Formal Specification

Hillel Wayne

When dealing with very complex systems, finding bugs becomes much more difficult. Although a wide variety of tools can help you, these tools primarily help identify why a bug has happened. We’ve done very little as an industry to help you avoid having the bug in the first place.

Why is that? In part, we are still used to thinking of bugs as faults in the code—uncaught nulls, off-by-one, and so on, but the subtlest and most dangerous bugs are problems with the design. They are cases when everything is locally correct but interact in a way that’s globally incorrect.

Consider mixing error retries and rolling deployment. The client’s initial request and first retry could be handled by different servers running different versions of the code. Any unexpected behavior wouldn’t be the fault of the client, server, or load balancer, but arise from the interplay among them.

Nobody has made a mistake. Every local component is doing exactly what we told it to do. Given the complexity of the system, it becomes difficult to understand the consequences of those actions at a global level.

The only way to deal with these is by the hard work of intelligent experts. Us. But just as we have tools to help us write code, we also have tools that help us write designs. One powerful technique is to write a software model of the system and ...