97 Things Every SRE Should Know

Chapter 27. Design Goals for SLO Measurement

Ben Sigelman

LightStep

When designing for SLO measurement, consider the goals of flexibility, testability, freshness, cost, reliability, and organizational constraints. Let me explain how to use them. You want flexible targets. That is, SLOs must be able to evolve over time. Sometimes this is simply to adjust an error budget to allow for more releases and faster product iteration.

Operators should be able to adjust the heuristics embedded in the SLIs, (for example, 25 ms [milliseconds] to 30 ms), success thresholds (95% of the time to 97% of the time), aggregation windows (over the past 30 seconds to over the past 7 days), and more, all without making code changes, redeploying software, or pushing new production configuration. The SLO performance history before and after the target revision should also be retained, with some way to see how each target has changed over time.

Next, consider testable targets. When adding a new SLO, we need both an SLI and an objective, or target. Crafting appropriate targets is often subtle and challenging. What’s the right error budget, given our reliability history? Percentile of latency to measure? Actual latency threshold? And, given our goal of flexible SLO targets, any time an SLO needs updating, these should all be reconsidered. To feel confident about our SLOs, backtest possible targets against ...