Security Monitoring

Chapter 1. Getting Started

It was mid-January 2003. Things were going well in my role as a network engineer supporting data center networks at Cisco. My team celebrated on January 21 when our site vice president powered off the last Avaya PBX; the Research Triangle Park (RTP) campus telephony was now 100% VoIP. We had just completed several WAN circuit and hardware upgrades and were beginning to see the highest availability numbers ever for our remote sites. Then, on January 25 (a Saturday at the RTP campus), the SQL Slammer worm wreaked havoc on networks around the world. Slammer, also known as Sapphire, targeted vulnerable MS-SQL servers using self-propagating malicious code. Security professionals surely remember the event well. The worm’s propagation technique created a potent denial-of-service (DoS) effect, bringing down many networks as it spread.

The only attribute distinguishing the Slammer worm from normal SQL traffic was a large number of 376-byte UDP packets destined for port 1434.^[1]

ISPs used ingress/egress filtering to block traffic, but by then it was too late to prevent system compromise; rather, it was a mitigation measure to protect the Internet backbone:

The Sapphire Worm was the fastest computer worm in history. As it began spreading throughout the Internet, it doubled in size every 8.5 seconds. It infected more than 90 percent of vulnerable hosts within 10 minutes.^[2]

The rate of replication and multitude of compromised systems on company networks began to saturate network links with propagation attempts. Network administrators saw this issue on some of the WAN links in the United States when their pagers began to light up like Christmas trees with utilization alerts, followed by link down Simple Network Management Protocol (SNMP) traps. Initially, the problem was thought to be related to a DS3 network card we had just replaced in one of our Southeast region WAN routers; however, as the issue appeared in other regional office WAN links, it became clear that this was not an isolated incident.

We had experienced the network problems caused by virus outbreaks such as Code Red (which attacked vulnerable Microsoft IIS web servers), but none approached the severity of network impact that Slammer did. A few Slammer hosts were able to generate enough traffic to take down WAN links, causing intermittent connectivity problems in our remote sites globally. Ultimately, a majority of the compromised systems were traced to unpatched lab servers. Identifying and mitigating these hosts was no easy task:

• Too few network intrusion detection systems (NIDSs) were deployed, and no one was responsible to view or follow up on alerts for infected systems.

• Network telemetry (such as NetFlow) and anomaly detection were insufficient to identify infected systems.

• There was no way to prioritize the response; the only data we had were IP addresses and DNS names of affected machines. We didn’t have contextual information such as “data center host” versus “user LAN host” versus “lab host.”

Over the next 48 hours, global networking teams identified infected systems using a manual process that involved deploying the recommended access control lists (ACLs) on remote WAN routers^[3] to block packets. Matches on the deny access control entries (ACEs) for UDP 1434 indicated an infected host at the site. We could not identify the source IP address that was creating the deny entries, as adding the “log” clause to the end of the deny ACE spiked the router’s CPU and drastically degraded network performance. The next step required network engineers to analyze switch port utilization in real time, searching for the infected host to disable its port. This manual process required substantial man-hours to address.

If we had implemented a few of the recommendations detailed in this book, our networking team could have contained the threat much more rapidly. A tuned NIDS deployment would have enabled us to locate the infected IP addresses immediately, prioritizing response based on their named network association (data center servers, lab hosts, or desktop systems, as you’ll see in Chapter 6). Even prior to the availability of the NIDS signature, we could have used NetFlow to identify infected hosts based on recognized traffic patterns, as we’ll discuss in Chapter 3. A prioritized, planned response would have occurred based on this information, with appropriate mitigation measures applied to the impacted systems. The IP information from NetFlow alone could have allowed for quick manual inspection of the router ARP tables and associated MAC-to-IP address mapping. Armed with that mapping, the network engineers could have quickly disabled ports on the access switches, shutting down worm propagation.

This book details infrastructure and frameworks that would have further helped when Nachi broke out several months later. Since we couldn’t see the future, however, Nachi created the same effect and was addressed with the same manual process as Slammer.

A Rapidly Changing Threat Landscape

We’ve heard it before: “gone are the days of script kiddies and teenagers out to wreak havoc just to show off.” The late 1990s and early 2000s produced a staggering number of DoS attacks. Malware, the engine for the DoS attack, has progressed from simple programs that attack a single vulnerability to complex software that attacks multiple OS and application vulnerabilities.

Let’s look at the description of the Nachi worm’s method of infection (circa 2003):

This worm spreads by exploiting a vulnerability in Microsoft Windows. (MS03-026)

Web servers (IIS 5) that are vulnerable to an MS03-007 attack (port 80), via WebDav, are also vulnerable to the virus propagating through this exploit.^[4]

Here’s information on a very popular virus from 2006 called SDBot:

The worm propagates via accessible or poorly-secured network shares, and some variants are intended to take advantage of high profile exploits:

WEBDAV vulnerability (MS03-007)

LSASS vulnerability (MS04-011)

ASN.1 vulnerability (MS04-007)

Workstation Service vulnerability (MS03-049)

PNP vulnerability (MS05-039)

Imail IMAPD LOGIN username vulnerability

Cisco IOS HTTP Authorization vulnerability

Server service vulnerability (MS06-040)

When it attempts to spread through default administrative shares, for example:

PRINT$

E$

D$

C$

ADMIN$

IPC$

Some variants also carry a list of poor username/password combinations to gain access to these shares.

Weak Passwords and Configurations

Several variants are known to probe MS SQL servers for weak administrator passwords and configurations. When successful, the virus could execute remote system commands via the SQL server access.^[5]

This more complex form of malware has components to make it persistent between reboots and to cloak itself from detection by antivirus programs. It even includes obfuscation techniques to prevent offline analysis! Many malware programs include a component to steal information from the infected system and relay it back to its creator, leveraging a remote control component (commonly called a botnet), which provides a vast array of capabilities to command the compromised system. Group all of these traits together—decentralized command and control structures (such as web-based or peer-to-peer [P2P] structures), and encryption and polymorphism (so that the malware can modify itself upon propagation to another system, evading detection by antivirus software)—and you can easily see why antivirus technology rarely lives up to its promise.

Failure of Antivirus Software

Hopefully, you no longer rely solely on antivirus software to detect and protect your end-user systems. Rather, a defense-in-depth strategy includes antivirus software, adding OS and application patch management, host-based intrusion detection, and appropriate access controls (we said “hopefully” ☺). If you are still relying exclusively on antivirus software for protection, you will be very disappointed. For example, in summer 2008, many of our employees received a well-crafted phishing campaign that contained a realistic-looking email regarding a missed shipment delivery from UPS:

-----Original Message-----
From: United Parcel Service [mailto:[e-mail]]
Sent: Tuesday, August 12, 2008 10:55 AM
To: [e-mail]
Subject: Tracking N_ 6741030653

Unfortunately we were not able to deliver postal package you sent on July the 21st
in time because the recipient's address is not correct.
Please print out the invoice copy attached and collect the package at our office

Your UPS

Attached to this email was a trojan that more than 90% of the 37 antivirus software programs were unable to detect. Table 1-1 shows the test results yielded by analysis of the trojan binary.

Table 1-1. Trojan binary analysis test results

Antivirus	Result	Antivirus	Result
AhnLab-V3	-	Kaspersky	-
AntiVir	-	McAfee	-
Authentium	W32/Downldr2.DIFZ	Microsoft	-
Avast	-	NOD32v2	-
AVG	-	Norman	-
BitDefender	-	Panda	-
CAT-QuickHeal	-	PCTools	-
ClamAV	-	Prevx1	-
DrWeb	-	Rising	-
eSafe	-	Sophos	-
eTrust-Vet	-	Sunbelt	Trojan-Spy.Win32.Zbot.gen (v)
Ewido	-	Symantec	-
F-Prot	-	TheHacker	-
F-Secure	-	TrendMicro	-
Fortinet	-	VBA32	-
GData	-	ViRobot	-
Ikarus	Win32.Outbreak.UPSRechnung	VirusBuster	-
K7AntiVirus	-	Webwasher-Gateway	-

As you can see from the test results, these antivirus products, which detect malware via “known bad” signatures, failed to identify the trojan. Such technology fails primarily because an insignificant change to the virus will make it undetectable by existing signatures. Vendors are improving their techniques—by including heuristic/behavioral-based detection, for example—but they still fall far short of providing “complete” system security. An excellent source for more information regarding viruses, their capabilities, and why they are able to hide from detection is John Aycock’s book, Computer Viruses and Malware (Springer).

The prevalence and advanced capabilities of modern malware should be reason enough to closely monitor for its existence in your network. If it isn’t, perhaps its use by Mafia-like organizations of criminals for profit via identity theft, extortion, and espionage is more convincing.

---

^[1] http://www.cert.org/advisories/CA-2003-04.html

^[2] http://www.caida.org/publications/papers/2003/sapphire/sapphire.html

^[3] http://www.cisco.com/warp/public/707/cisco-sn-20030125-worm.shtml

^[4] http://vil.nai.com/vil/content/v_100559.htm

^[5] http://vil.nai.com/vil/content/v_139565.htm