97 Things Every SRE Should Know

Chapter 51. When SLOs Attack: Pathological SLOs and How to Fix Them

Narayan Desai

Google

SLOs are a wonderfully intuitive concept: a quantitative contract that describes expected service behavior. These are often used to build feedback loops that prioritize reliability, communicate expected behavior when taking on a new dependency, and synchronize priorities across teams when problems occur, among other use cases.

However, SLOs are built on an implicit model of service behavior, with a raft of simplifying assumptions that don’t universally hold—assumptions such as the independence of requests, even distribution of errors, and the equality of all requests. These assumptions make SLO rules of thumb fall apart with real-world services. Understanding where and how these assumptions break down is critical: cases when SLOs inadvertently send us in the wrong direction.

Consider error budgets: a number or percentage of failures over a time interval. These errors could occur in a short period or at a low rate over a long time. They could be distributed across all users or focused on a few. Individual users could have low or 100% error rates. All these factors color how outages will be perceived and what kinds of effects they have on users.

Further, how best to deal with catastrophes? Many service providers try to incorporate bad days into SLO promises; however, some bad days are very, ...