Hands-On Infrastructure Monitoring with Prometheus

Percentiles

We just learned how to extract the percentage of requests being served under a given latency, but what if we need to understand the latency of a given percentile of requests?

To obtain, for example, the 95th percentile, we can use the histogram_quantile function by defining the quantile (in this case, 0.95), and then feed it the query expression that represents the set of data we're interested in—the average rate of increase for each of the buckets in the request duration histogram during a rolling time window. If we want the global latency of the service, instead of per instance/pod/handler, we need to apply the sum() aggregation:

histogram_quantile(0.95, sum without (instance, pod, handler) (rate(prometheus_http_request_duration_seconds_bucket[5m]))) ...