Practical Monitoring

Chapter 4. Statistics Primer

Statistics is an undervalued topic in the world of software engineering and systems administration. It’s also misunderstood: many people I’ve spoken to over the years are operating on the misapprehension that “rubbing a little stats on it” will result in magic coming out the other end. Unfortunately, that isn’t quite the case.

However, I am happy to say that a basic lesson in statistics is both straightforward and incredibly useful to your work in monitoring.

Before Statistics in Systems Operations

Before we get into the statistics lesson, it’s helpful to understand a bit of the background story.

I fear that the prevalence and influence of Nagios has stifled the improvement of monitoring for many teams. Setting up an alert with Nagios is so simple, yet so often ineffective.¹

If you want an alert on some metric with Nagios, you’re effectively comparing the current value against another value you’ve already set as a warning or critical threshold. For example, let’s say the returned value is 5 for the 15m load average. The check script is going to compare that value against the warning value or critical value, which might be 4 and 10, respectively. In this situation, Nagios would fire an alert for the check breaching the warning value, which is expected. Unfortunately, it isn’t very helpful.

As so often happens, systems can behave in unexpected (but totally fine) ways. For example, what if the value crossed the threshold for only one occurrence? What ...