Hands-On Infrastructure Monitoring with Prometheus

Sum-of-rates versus rate-of-sums

The previous point might seem obvious, but when the complexity of the queries built starts to increase, it's easy to make mistakes. A common example of this is trying to rate a sum of counters instead of summing rates. The rate function expects a counter, but a sum of counters is actually a gauge, as it can go down when one of the counters resets; this would translate into seemingly random spikes when graphed, because rate would consider any decrease a counter reset, but the total sum of the other counters would be considered a huge delta between zero and the current value. In the following diagram, we can see this in action: two counters (G1, G2), one of which had a reset (G2); G3 shows the expected aggregate ...