I think we ought to take the men out of the loop.
— “War Games,” 1983
In the Introduction and Chapter 2 we discussed some of the challenges of notifications. The primary purpose of this chapter is to make notifications useful: to send them at the right time and put useful information in them.
Sending too many notifications is the monitoring equivalent of “the boy who cried wolf”. Recipients will become numb to notifications and tune them out. Crucial notifications are often buried in the flood of unimportant updates.
And even if an notification surfaces, it is not always useful. In Chapter 2 we saw a stock Nagios notification.
PROBLEM Host: server.example.com Service: Disk Space State is now: WARNING for 0d 0h 2m 4s (was: WARNING) ...