Chapter 1. Introduction
This book is about building a network-based intrusion detection system (NIDS) based on the open source application called Snort. Snort got a modest start as the open source project of a software engineer names Martin Roesch (who incidentally was the lead engineer in the development of an IDS solution for GTE). Snort is now a high-performance, full-featured solution that provides competition for some very expensive commercial solutions (and surpasses many).
A context for the use of an NIDS solution is established by examining the challenges confronting a network administrator with regards to security. New technologies are making it easier for remote users and partners to access the insides of the network, bypassing perimeter security entirely. A new breed of Internet worm is attacking from a variety of directions—through email, across the network, and even across virtual private network (VPN) connections. Hacker communities are creating tools that make attacking a network much easier. This gives rise to "script kiddies,” who download an attack tool and penetrate an organization’s network—all without knowing how the tool they are using works or the effect it will have on the target system.
In the old days (two years ago or so), a firewall was most of what an administrator needed to protect a network from attack. It was easy to establish where your network ended and the Internet began. Technological advances and decreasing costs for wide area network technologies have eroded this concept of a perimeter. VPNs have all but replaced conventional dial-up modem pools. Most users have high-speed DSL or Cable Modem service, and the VPN makes the user feel like he’s sitting at his desk. Some VPNs use an appliance that sits on the perimeter of the network and has the capability of controlling how the network is used remotely. While this is a boon for telecommuters, it is a real risk for most networks. A virus or worm-infected system on the user’s home network suddenly has unfettered access to the inside of your network. That high-speed highway into your network can allow rapid propagation of an aggressive worm.
Connections to business partners used to be an expensive proposition and were only for the most well-to-do organizations. Dedicated T1 links are expensive. With less expensive network options (not to mention network-to-network VPN connections), this cost has decreased significantly. This allows many organizations to connect their network to yours—sometimes directly into the internal network. Without real precautions in place, security problems on the partner networks quickly become security problems on your network—very often undetected until much damage is done. Whether you trust your partner to that extent is another matter.
When deploying troops in a theater of war, a general has to consider all the ways an enemy may attack: by land (either at the front line, or a commando raid behind the lines), by sea (surface ships or submarines), or by air (helicopters, fighters, bombers, missiles, or artillery). The general has to deploy defenses against all potential vectors of attack. He doesn’t just trust the trenches at the front line for all his security. He will deploy troops to the front line, as well as at high-value assets behind the lines. He will deploy a variety of anti-submarine and anti-surface ship defenses. He will deploy a variety of anti-air assets to protect against the various air threats. This concept of multiple overlapping defensive measures is known as defense-in-depth .
A similar system can be applied to network security. Instead of trusting the eroding value of perimeter defenses (firewalls) for all of our security, we turn to other mechanisms. We configure systems according to industry-accepted best practices (disable unnecessary services, keep software updated, run antivirus software). We establish a system to securely aggregate our system logs in one place (and we monitor those logs for anomalies). We segregate our network to control access to important machines and to “wall-off” partner and remote connections. We utilize strong authentication and authorization practices. And finally, we take steps to detect and prevent intrusions (preferably attempted intrusions) on our network and on our systems. We also try to do this with limited budgets and limited time. In the real world, the general is trying to protect against lost real estate. In the network world, the administrator is protecting against downtime and data loss. I won’t beat the analogy to death. The main thing to remember is not to trust a single component of your security framework for all your security. If you are able to, apply security as close to the thing you are trying to secure as possible. These steps will help you stop at least 80% of attacks. Intrusion detection should catch the remaining 20%.
Detecting Intrusions (a Hierarchy of Approaches)
Intrusion detection is simply trying to detect the signs of a network intruder before damage is done, a service denied, or data lost. This can be done through the use of a variety of mechanisms. Properly configured systems generate system logs that keep track of services, users, and data. These logs very often show traces of suspicious (or downright nefarious) activity. The problem is that these logs often have a lot more information in them than a security administrator is interested in. It is important to consider system log review as a basic intrusion detection mechanism, though. Many times the system logs show their value in a forensic analysis after the fact.
The next layer of intrusion detection (and prevention) is automated tools, commonly referred to as host-based intrusion detection (HIDS). HIDS tools include antivirus software, personal firewalls, NIDS installed on the individual hosts, and a new breed of software (intrusion prevention systems) that protects system memory against buffer overflow attacks or enforces security policies. Many products are a hybrid mix of these solutions (a personal firewall/antivirus product, for example).
The final layer of intrusion detection is NIDS.
What Is NIDS (and What Is an Intrusion)?
On a basic level, network intrusion detection is exactly what it sounds like: the process of determining when unauthorized people are attempting to break into your network. Keeping those attackers out or extracting them from the network once they’ve gotten in is a different problem. Obviously, keeping intruders out of your network is a meaningless task if you don’t know when they’re breaking in.
Detecting unauthorized connections is a good start, but it is not the whole story. Network intrusion detection systems like Snort are great at detecting attempts to login to your system, access unprotected network shares, and things like that. But there are other kinds of intrusion that are not as clear-cut as an outsider walking past the receptionist at the front desk and sitting down at a computer. Is a denial of service attack—one that operates by sending a carefully crafted sequence of packets to a network server and ultimately crashing it—an intrusion? No one has literally gained access to your machine’s physical resources. However, bandwidth, CPU time, and hard-drive space on your IDS are all consumed by the attack. Denial of service is considered a successful attack because it occupies resources that would have been employed somewhere else. Does someone probing your networks with port scans or pings constitute an intrusion? Perhaps not, but it is a sign that she may soon start doing something more hostile. So we also consider probing an intrusion, and expect our intrusion detection system to warn us whenever things such as these happen.
Generally speaking, an intrusion detection system like Snort scans network traffic looking for suspicious activity based on the signatures of bad packets. You are probably already familiar with tools like tcpdump and ethereal, which display all the traffic flowing on your network within a specific subnet. An intrusion detection system is essentially an automated tcpdump, a packet sniffer that sniffs in the background and does not require you to watch or analyze the traffic yourself. Tools like ethereal work well for debugging; for instance, when you have to look at each packet to figure out what might be wrong. But on any real network, there is just too much traffic to watch for suspicious activity. That is what computers are good for: doing a very boring job repetitively, and alerting you when something interesting comes along.
An IDS watches the packets traversing your network and decides whether anything is suspicious. How does it know what is suspicious? Snort bases its analysis on the signatures of bad packets: essentially, a list of descriptions of the types of packets that indicate the system is under attack or a successful attack has already taken place. For example, if you receive an ICMP packet that is abnormally large, you may infer somebody is trying the antiquated ping of death attack against a host on the network. If you see fragmented packets that are extremely short, you may also infer that somebody is trying one of the many attacks that rely on fragmentation to sneak by firewalls. Snort (and other intrusion detection systems) comes with thousands of signatures, based on attacks that have been observed “in the wild.” The list grows longer every day and updates are constantly posted to the Snort web site. Part of the job (and one that is managed nicely by the tool we will soon discuss) is keeping your signature list up-to-date.
Snort and other intrusion detection systems thus provide an important first line of defense against attacks. If an intruder manages to login to your network server, you might be able to find the evidence in system logs, although a smart cracker would delete your logfiles. The host intrusion detection system watches for unauthorized activity on an individual system. If someone manages to compromise the same server using a fragmentation attack, you might be able to figure out what happened after the fact by looking at logs, but you might not—and at that point, it is too late.
While it is too optimistic to talk about “real-time” intrusion detection, it is extremely important that an IDS detect attacks early, before any damage can be done, and that it send notifications to you and to a secure database. We discuss “invisible” or stealthy methods of logging Snort’s warnings and alerts to a database elsewhere. If you can head off an attack, so much the better—but even if you cannot, an IDS might be the only way to figure out what happened and prevent it from happening again.
The Challenges of Network Intrusion Detection
The benefits of detecting an intrusion as early as possible are undeniable. But it is important to deploy an IDS with realistic expectations. There are some real challenges in installing, maintaining, and interpreting the output from an intrusion detection system.
A potential intrusion detection administrator needs a good knowledge of the environment into which she is introducing NIDS. What is the network layout? This information helps determine the positioning of the sensors and also may help determine which mode of operation should be used. What kinds of systems are in the environment? Windows? Unix? What services are the systems providing? Email? Web services? How is encryption used in the environment?
A good understanding of how systems communicate on the network is very important in interpreting the output of the NIDS sensors. Without knowing the makeup of a TCP packet, an alert specifying a problem within a packet will only cause confusion. If you are not familiar with network sniffing tools like tcpdump and ethereal, spend some time watching the traffic on your network. Review the contents of Chapter 2 to help you interpret the results. Only good can come from this time spent watching and learning how things talk and move around your network. Without this background, the job of determining what is really something to worry about—as well as tuning out unneeded rules and features—is very difficult.
Very often (and especially before tuning), when Snort sends you a warning that something suspicious is happening, there is nothing really serious going on. Any NIDS is going to generate a lot of false positives, warnings that someone or something is launching some form of attack, when in fact nothing is happening. You may be able to minimize false positives, but you cannot entirely eliminate them. Furthermore, the more false positives you receive, the more likely it is that Snort is missing an actual attack or subversive intrusion attempt. It is up to you to figure out an acceptable level of risk. Do you really want to be notified about every port scan? About every unauthorized attempt to mount a Windows share? Even on a home network this can quickly drive most sane administrators crazy.
There is no perfect solution. There’s an easy way to guarantee an attack is never goes unnoticed: flag every incoming packet as suspicious. That is obviously not realistic. You won’t have to worry about missing a potential attack, but the flood of false positives will be overwhelming. At the other extreme, you could tune out the majority of alerts and turn off most of the features of Snort. You won’t have many false positives, but you’ll also miss many of the real dangers. You must find a happy medium and decide just how many alerts you are willing to tolerate for the sake of your network. The process of reaching this compromise can only be accomplished over an extended period of time, by fine-tuning Snort and viable signatures and enabling or disabling features within the Snort sensors themselves.
A common phrase I use when talking to clients about deciding to deploy an IDS is, “If you don’t do basic system log review on a regular basis, an IDS is just going to generate more logs for you to ignore.” As discussed earlier, system logs are the first line of defense in intrusion detection. Reviewing system logs yields great benefits in learning how your systems function and in determining the health and well-being of your systems. An IDS only provides value as a component in a defense-in-depth strategy. Do not lay all security responsibility on your IDS installation.
When deciding to embark on a Snort installation (or any other NIDS solution, for that matter), understand that there is some significant work that needs to be done on the frontend. None of it is particularly difficult, just time-consuming and detail-oriented. A common misconception is that once the NIDS sensors are deployed and initially configured, and the central management console is built and reporting, the administrator can throw a dust cloth on it and walk away. Snort is a signature-based NIDS. Signatures need to be updated periodically to keep up with the latest exploits and attack methods. They also need constant tuning to eliminate false positives and allow for changes in your network environment. These tasks are not overwhelming, but not allowing time for them greatly diminishes the value of the NIDS deployment.
Why Snort as an NIDS?
Snort represents a cost-effective and robust NIDS solution that fits the needs of many organizations. This book should be all you need to get Snort installed, configured, tuned, and alerting accurately in your environment. Snort is covered from initial configuration to ongoing maintenance. Strategies are revealed to make Snort useful for a home office or a large corporation with a dedicated and experienced network security staff. The approach is one of attempting to derive reasonable approaches to the issues at hand. I try hard not to be a zealot.
Snort does not stand by itself as the beginning and end of a security framework for an organization. It is part of an overall defense-in-depth strategy that incorporates security in all aspects of a network. Whether Snort is an important and significant contributor relies on strong planning and an ongoing dedication to the care and feeding of your NIDS.
There are a wide variety of choices in the area of intrusion detection. Digging through the propaganda generated by the various marketing departments is not easy. Even the definition of intrusion detection is murky, often moving from one solution to another. To cut through the noise, consider the following:
Open source software is hard to beat on price. To be sure, very often such software can be more difficult to operate. Snort is one of the more mature open source packages out there and competes with any commercial product for return on investment. There is the occasional C-level executive that will throw out an open source solution because there is no one to call when it breaks. With mainstream acceptance of open source solutions increasing constantly, this is less often a problem. For those who cling to this thinking, there are several commercial products that use Snort as their core technology. Chief among these is Sourcefire, an organization at the forefront of Snort development and implementation. Sourcefire was started by a fellow named Martin Roesch, now the CTO (does that name sound familiar?).
- Stability, speed, and robustness
Since very early on, one of the main goals of Snort’s developers was to keep it lightweight, fast, and lean, in order to keep up with ever-increasing network bandwidths. Since it is not a new solution, bugs are virtually nonexistent. A Snort instance crashing is almost unheard of. I personally have a Snort installation that watches sustained 450 Mbps of bandwidth using a cluster of six sensors. The only time Snort is down is during a planned maintenance window to upgrade signatures or move to a new version. This demonstrates not only Snort’s stability, but also its ability to be adapted to very demanding environments (see Chapter 13).
- The preprocessors
In Chapter 5, I go into great detail on the inner workings of the Snort preprocessors. For the moment, let me just say that the preprocessors massage the network data flow in real time to increase the chances of a signature noticing a malicious packet. The incredibly complex ways that computers can communicate and be used on a network presents a real challenge. The preprocessors act as interpreters for the Snort detection engine. Another real strength of the preprocessors is their ability to defeat many IDS evasions techniques. Chapter 4 discusses the ways that attackers go after your systems and also the ways they try to trick, hide from, or simply overwhelm your IDS defenses.
Snort is very flexible in the ways it can be deployed. Chapter 4 through Chapter 8 detail the ways that Snort can be used, from a simple network sniffer to a true gateway IDS that kills a dangerous network conversation in its tracks. Because you can customize existing signatures or write your own custom rules, Snort can adapt to almost any situation.
There are a number of applications that can act as central monitoring and alerting consoles. I talk about several, concentrating on ACID and SnortCenter. There are also a number of community contributed scripts and plug-ins that extend Snort’s functionality—allowing syslogs to be parsed and alerted from, and another allowing the dynamic creating of access control lists on Cisco routers, for example.
- Industry support
Particularly with the advent of several commercial versions of Snort, many security industry watchdogs include Snort signatures in their security announcements (CERT and SANS, to name two). The Snort open source community is very active keeping signatures up to date. When worms are ravaging the Internet and there are constantly new variants, there are sometimes updates multiple times a week. The Snort mailing lists are a fantastic resource for people who are trying to run Snort or write their own signatures.
Sites of Interest
SecurityFocus IDS Page
The SANS Institute
The NSA Security Guides