Site Reliability Engineering, 2nd Edition

Appendix G. Modern Observability: Statistics and ML

This appendix provides a deeper look into the statistical and machine learning models we use for the intelligent alerting systems described in Chapter 9. While the chapter focuses on the strategic shift away from static thresholds, here we explore the technical underpinnings of how we create alerts that are both precise and adaptive.

A Statistical Approach for Error Rates

To move away from static alerting thresholds, we see an opportunity to apply a more rigorous statistical approach. Consider one of the most common signals: error rates. For any given request, the outcome is binary—it either succeeds or fails. This scenario isn’t just a random stream of events; it is a classic statistical pattern that we can model with a binomial distribution.

The first step is to teach our system what “normal” looks like for each individual service. Instead of a one-size-fits-all threshold, we establish a dynamic baseline error probability by analyzing the last two weeks of traffic. Without diving too much into mathematical formalisms, we use Wilson’s confidence interval to describe the expected distribution of errors, and if we go above the maximum expected error range, we alert (many statistical methods could be used here with similar results, but we have found this one to be robust and easy to implement).¹

This approach is naturally volume-aware. For a low-traffic service, the acceptable range of errors is wider, preventing alerts from a ...