Security Monitoring

Monitor the Monitors

If you’ve worked in information security for long, you’ve noticed a phrase emerge over the past few years: “watching the watchers”. The idea is to keep an eye on your privileged users—the ones maintaining the systems for common users. We must apply the same concept here: to make sure the systems used for conducting security monitoring are reliably maintained. Without that assurance, critical security events may be lost or deliberately suppressed.

To effectively monitor the health of your monitoring system, you must first set a benchmark for “normal” activity. As Æleen Frisch writes in Essential System Administration: “As with most of life, performance tuning is much harder when you have to guess what normal is. If you don’t know what the various system performance metrics usually show when performance is acceptable, it will be very hard to figure out what is wrong when performance degrades. Accordingly, it is essential to do routine system monitoring and to maintain records of performance-related statistics over time.”^[56]

You’ll use this benchmark to compare recent activity and determine whether anything has changed, and whether that change requires attention. For example, if you know that your NIDS normally sustains 400 Mbps of traffic, a sudden, sustained drop to 5 Mbps should cause you to take note and look for problems on the network or the devices. If you normally receive 150 messages per hour from your Windows domain controller, you’ll want to be notified ...