Network Security Hacks

Chapter 5. Monitoring and Trending

Hacks #61-66

While the importance of reliable system logs can’t be overestimated, logs only tell part of the story of what is happening on your network. When something out of the ordinary happens, the event is duly logged to the appropriate file, where it waits for a human to notice and take the appropriate action. But logs are valuable only if someone actually reads them. When log files add to the deluge of information that most network administrators already wade through each day, many log files may go unread for days or weeks. This situation is made worse when log files are clogged with irrelevant information. For example, a cry for help from an overburdened mail server can easily be lost if it is surrounded by innocuous entries about failed spam attempts. All too often, logs are used as a resource to figure out “what happened” when systems fail, rather than as a guide to what is happening now.

Another important aspect of log entries is that they only provide a “spot check” of your system at a particular moment. Without a history of what normal performance looks like, it can be difficult to tell the difference between ordinary network traffic, a DoS attack, and a visitation from Slashdot readers. While you can easily build a report on how many times the /var partition filled up, how can you easily know what usage looks like over time? Is the mail spool clogged due to one inconsiderate user, or is it part of an attack by an adversary? Or is it ...