Network Security Through Data Analysis, 2nd Edition

Chapter 3. Sensors in the Network Domain

This chapter is concerned with the data generated by network sensors. These are sensors that collect data directly from network traffic without the agency of an intermediary application, making them service or host domain sensors. Examples include NetFlow sensors on a router and sensors that collect traffic using packet capture, most notably tcpdump. This also includes middlebox services such as VPNs or NATs, which contain log data critical to identifying users.

The challenge of network traffic is the challenge you face with all log data: actual security events are rare, and data costs analysis time and storage space. Where available, log data is preferable because it’s clean (a high-level event is recorded in the log data) and compact. The same event in network traffic would have to be extracted from millions of packets, which can often be redundant, encrypted, or unreadable. At the same time, it is very easy for an attacker to manipulate network traffic and produce legitimate-looking but completely bogus sessions on the wire. An event summed up in a 300-byte log record could easily be megabytes of packet data, wherein only the first 10 packets have any analytic value.

That’s the bad news. The good news is that network traffic’s “protocol agnosticism,” for lack of a better term, means that it is also your best source for identifying blind spots in your auditing. Host-based collection systems require knowing that the host exists ...