Site Reliability Engineering, 2nd Edition

Appendix G. Modern Observability: Statistics & ML

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be Appendix G of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

This appendix provides a deeper look into the statistical and machine learning (ML) models we use for the “Intelligent Alerting” systems described in the Observability and Monitoring chapter. While the chapter focuses on the strategic shift away from static thresholds, this appendix explores the technical underpinnings of how we create alerts that are both precise and adaptive.

A Statistical Approach for Error Rates

To move away from static alerting thresholds, we see an opportunity to apply a more rigorous statistical approach. Consider one of the most common signals: error rates. For any given request, the outcome is binary—it either succeeds or it fails. This scenario isn’t just a random stream of events; it is a classic statistical pattern that we can model with a binomial distribution.

The first step is to teach our system what “normal” looks like for each individual service. Instead of a one-size-fits-all threshold, we establish a dynamic ...