Site Reliability Engineering, 2nd Edition

Appendix F. The Mathematics of Latency Alerting

Formally, assume that there is a set of user-generated workloads $w\in W$ that are independently run on a service. Additionally, there is some metric $m:W\to R$ that measures the performance of the workload on the service, such as latency.

You partition the workloads $W$ into $N$ distinct and disjoint cohorts $C_i$, where $1\le i\le N$, such that ${\cup }_{i\le N}{C}_i=W$.

Because each workload is (essentially) unique—they may write to different files, compute on different data, be executed on different clusters, etc.—you must cluster workloads in a fuzzy way that captures human expectations of equivalence. Cohorts need to be fine-grained enough to separate workloads with distinct behavior and coarse enough to maintain a reasonable signal-to-noise ratio. A cohort $C_i$ is considered “meaningful” if it meets a minimum cardinality threshold $B$ (e.g., $B\ge 10,000$), and its variance is small relative to its mean:

$$CV(C_i)=\frac{\sqrt{Var(m(C_i))}}{\overline{m}(C_i)}\le K$$

where:

$$K\le 0.5$$

The goal is to maximize the number of workloads that fall into these meaningful cohorts; typically, we aim for the union of meaningful cohorts to cover at least 65% of the total workloads, $W$.

Once cohorts have been defined, compute baseline performance expectations. For simplicity, you can assume that for a given cohort $C_i$, the performance of its workloads is independent and identically distributed, following an underlying normal distribution:

$$m(w)~N({\mu }_i,{\sigma }_i^2)\text{ for }w\in C_i$$

In practice, we use a 30- to 60-day baseline dataset to estimate ...