Site Reliability Engineering, 2nd Edition

Appendix F. The Mathematics of Latency Alerting

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be Appendix F of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Formally, assume that there is a set of user-generated workloads $w\in W$ that are independently run on a service. Additionally, there is some metric $m:W\to R$ which measures the performance of the workload on the service, such as latency.

You partition the workloads $W$ into $N$ distinct and disjoint cohorts $C_i$, where $1\le i\le N$, such that ${\cup }_{i\le N}{C}_i=W$.

Because each workload is (essentially) unique—they may write to different files, compute on different data, be executed on different clusters, etc.—you must cluster workloads in a fuzzy way which captures human expectations of equivalence. Cohorts need to be fine-grained enough to separate workloads with distinct behavior; they also need to be coarse enough to maintain a reasonable signal-to-noise ratio. A cohort $C_i$ is considered “meaningful” if it meets a minimum cardinality threshold $B$ (e.g., $B\ge 10,000$), and its variance is small relative to its mean:

$$CV(C_i)=\frac{\sqrt{Var(m(C_i))}}{\overline{m}(C_i)}\le K$$

where

$$K\le 0.5$$

The goal is to maximize the ...