Naturally, there are any number of mistakes that can be made in setting up an incident response team. While we couldn’t possibly hope to capture all of them here (our editors have set a maximum length for this book!), this section describes some of the more common pitfalls and how to avoid them.
In commercial enterprises, business comes first. Even in noncommercial enterprises, this is true to some degree. One of the cardinal rules of incident response is never to hinder the company’s ability to do business. That is, the incident response team should understand that it plays a supporting role to the company’s business activities. As such, it is important for the team to put the needs of the business first. A common mistake for incident response folks is that they focus exclusively on the technical aspects of an incident and its resolution, without fully appreciating the business situation. It is not enough to say, for example, that a web site has been compromised by Attacker X using Tools Y and Z from Remote School A; instead, it is also vital to understand what business process has been impacted by the web site’s incident, and what the financial impact is.
For example, assume that a high-traffic e-commerce web server of a major dot com company is compromised by a vulnerability. Shutting the server down in the middle of the work day for the investigation may be the best course of action from the perspective of the incident response team, but what is best for the company from the CEO’s perspective? Shutting the server down in the middle of the day probably means significant lost revenue, and that’s not good for a company’s health. A happy medium might be to bring the server down during a scheduled maintenance period or when the site has its slow period and few visitors. Only by appreciating the business consequences of a situation can an effective response be formulated.
Incident statistics can be the bane of a team’s existence. They are tedious to keep, tabulate, and analyze, and are often difficult to quantify -- at least, in the absence of an acceptable set of community definitions and standards. They can be downright infuriating! To illustrate this point, imagine an incident in which a single PC becomes infected with a PC virus that was inadvertently released through an email message. The PC user is quickly alerted to the problem and uses the company’s authorized antivirus software to remove the virus. Most teams would call this one incident (since it affected a single system.) Now, imagine another incident in which an intruder breaks in to the company network and steals or alters information on computers in dozens of business units across the company. Is this one incident or is it in fact dozens of smaller incidents? What if the virus had spread to those same dozens of business units (e.g., Melissa) and each one had reported “its” incident independently to the incident response team; now how many incidents is it? More importantly, how do you describe these differences and the nuances to the company’s board of directors?
Obviously, the issue isn’t trivial. But many teams are expected to produce periodic statistics. After all, good stats serve many useful purposes. They help the company quantify and understand how bad the security problem is. They help the incident response team justify its own existence and resource levels. The difficulty is in defining what incidents are useful and reporting them in a clinical, but still useful manner.
The key to success is to start with a set of metrics and definitions (such as knowing what constitutes an incident and the impact of such an incident) and then be fanatical in following those standards. This is an area where it is best to seek opinions and guidance from other teams as much as possible, and then clearly define and articulate the standards for your own organization.
For example, during calendar year 2000, one of our client companies had the following major incidents that required incident response resources, shown in Table 4-1.
Table 4-1. Incidents and responses
Closely related to the issue of statistics gathering and reporting is the issue of customer confidentiality. Depending on the community that your team serves, it may be helpful or even necessary to offer some degree of customer confidentiality surrounding an incident. Few business managers want their business units mentioned by name in an incident report that goes to the senior executives, especially if the incident was handled inappropriately by the business unit or the team. That is a level of scrutiny that most people will seek to avoid, and the way that they will avoid it is never (again) to report an incident to the response team who ratted them out to senior management, no matter how bad the next incident they encounter is. Clearly, that is a situation that serves no one in the long run and should be avoided at all cost. One way of avoiding it is by empowering the response team to offer confidentiality to its customers. The response team is free to report incidents and statistics to the appropriate people, but the actual names of the business units involved in the incidents may be left out.
One such case from our days in the United States Department of Defense demonstrates how important customer confidentiality is in the incident response process. While running the operations of their incident response team, we were able to “sanitize” the statistics that we reported to senior DoD management by leaving out the names of the affected sites. Instead, we would report the total number of Army, Air Force, Navy, and Marine incidents, but not the actual installations that were affected. The raw, unsanitized data only went to the commanders of the affected sites. This met the operational needs of the commanders to recover from the incident, as well as the strategic, statistical needs of the senior management that needed to have an understanding of the overall size and scope of the problem. More importantly, the individual sites knew that they could report an incident to us without fear of senior management reprisal. As a result, we had a great deal of repeat customers who called us routinely to ask for guidance and recommendations.
Although this type of customer confidentiality won’t work everywhere, we strongly caution any response team planning on reporting incidents to the senior executives to carefully consider customer confidentiality.
Simply put, it can be extremely difficult to get the sufficient funding and staffing necessary to put together an incident response team. This is particularly true because information security, in general, plays a supporting role to most companies’ real business (unless your business happens to be information security!). For most companies, security functions are not revenue-generators; they are revenue consumers that leech away the company’s profits. That is often a tough sell for many companies, and suggests that many security organizations do not have adequate funding and staffing. While we certainly can’t change that, we do have some suggestions on how to maximize the team’s effectiveness. First, the team’s manager must be aggressive, assertive, and confident in presenting compelling business arguments in favor of supporting the team. No opportunity should be left unexplored. Next, depending on the funding structure you use to support the team, consider getting the business units the team serves to pay for the services they receive. This can be a politically charged topic at many organizations, so proceed cautiously! Also, consider a staff augmentation plan for handling large or multiple simultaneous incidents. In many, if not most companies, augmenting the staff through a matrix and/or contractor assistance can be highly cost effective while still providing the service the company needs during a crisis.
If funding and staffing are inadequate, the one thing to avoid like the plague is adopting a fatalist attitude (when you know that all the lifeboats are gone.) Nothing spreads like, or is more destructive to an organization than negative attitudes, especially from the team’s management. While we recognize this next statement to be highly controversial, if staffing and funding are such that it is impossible to succeed, then perhaps the company is not ready to have an incident response team, and it is time to move on to something else.
One of the most common and costly mistakes made by incident response teams is to enter into an adversarial relationship with the divisions or business units they are trying to support. While this isn’t usually done on purpose, one of the causes we’ve seen is that business units often view incident response teams as internal auditors looking for system weaknesses within the fiefdoms of the various business units. We cannot emphasize more the need for an incident response team to support its customers in every way; adversarial relationships are the kiss of death. The customers must trust the team implicitly and know that the team is always going to put the needs of the customer first.
Incident response is a customer service business. The only way an incident response team will remain in business, regardless of its funding structure, is to have a customer service attitude. That attitude needs to start with the senior management and must work its way to the employees who talk to the clients every day. If the people interacting with the clients are not service-driven, then the team is almost certainly doomed to failure.
The best ways to avoid that negative situation are: be highly selective in hiring the right people, train your staff on the basics of customer service, and always set the right example as computer security professionals -- as model computer users, adherents to established security policies and procedures, with a high degree of objectivity and integrity to customers. This is probably even more true in incident response than typical customer service organizations, because incident response teams deal with people in crisis on a routine basis. Never let their problems become routine; although they might seem like routine matters to you, they certainly are anything but routine to the person experiencing the emergency.
Several things can be done to avoid an adversarial relationship with the customers. First, if feasible, offer the customers a degree of confidentiality. No one likes to have his dirty laundry aired publicly within the company or to the outside world. Second, be responsive to the customer around the clock, all year. Third, ensure that the entire staff has a customer service attitude and that the customers always feel they have gotten more than what they expected.
You’re probably thinking, “Wait a minute, I’m in incident response, not customer care!” You’re correct that you are in the incident response line of work, but so is your doctor, paramedic, and health-care worker. You can be an outstanding doctor but if you have a lousy bedside manner, your patients might not open up to you as honestly as you like, nor give you enthusiastic endorsements and recommend you to all their friends. The ideal incident responder is technically competent and knows all the moving parts, but at the same time is compassionate and has outstanding people skills. That’s the difference between a good responder and a great responder.
Remember the customer service model in Chapter 3? Any customer who asks for help should be given direct assistance by the person who was approached by the customer. Under no circumstances should a customer be referred or passed off to another employee without a “hard handoff” by the person providing the assistance. On each of the teams we have worked on since then, that has been declared as our standard of customer support excellence. Sure, that means extra work for the employees and the additional overhead of holding all our clients hands, but it’s vital to ensuring that the customer gets the most out of an otherwise unpleasant experience.
Any interaction with the response team, even during difficult situations, should be as positive and honest as possible. Incidents seem to occur at the worst times. With the job of an incident responder comes the responsibility to support your customers whenever they call for help, not just during posted business hours.
Handling information that might need to be used as evidence requires particular attention to detail. A seemingly small mistake in the way the prospective evidence is handled can result in a costly disaster if a defense attorney, for example, were to cast doubt in the information’s integrity due to the flawed procedures. Not a good way to insure a lengthy career as an incident responder! Consult your General Counsel as well as law enforcement in your area for guidelines on evidence handling procedures. In the United States, the Department of Justice has a set of Federal guidelines for evidence handling at http://www.cybercrime.gov; this is a good starting point, depending upon your location, laws, and policies. There are also commercial training programs from companies like New Technologies in Gresham, OR, that focus on computer forensic training for commercial and law enforcement clients.
Above all, be sure to document your evidence-handling procedures prior to exercising them. The team’s staff needs to be thoroughly trained on how to handle evidence and maintain chain of custody, and the team leader needs to be fanatical in ensuring that the written procedures are always followed during an incident. It is all too easy to make simple mistakes in the heat of battle that can lose a prosecution in court for you.
Every country, state, and company has its own regulations on employee privacy, and it seems that individual privacy concerns are getting more attention than ever before. Technical processes that may seem to be perfectly reasonable while responding to an incident may, in fact, be in violation of one or more local policies. Similar to incorrectly handling evidence, violating an employee’s -- or even an outside intruder’s -- privacy can explode into a public relations nightmare, or even a civil law countersuit by the accused. To make things worse, the jurisdiction of a computer crime can be nebulous at best, something that law enforcement has been grappling with for over a decade.
Imagine an intruder using the Internet to break into a company. The intruder is physically located in Missouri and rides the Internet to an island in the Caribbean, and from that tropical cyberparadise, returns to the U.S. to break into a system in Nebraska. Was a crime committed in Missouri, the Caribbean island, or Nebraska? What if the network connections enroute took the intruder’s data packets through five other states and three countries? Now where was the crime committed? The answer to that depends on several factors. It depends, among other things, on which law enforcement officials the victim is able to convince to investigate the problem -- which state or country’s laws would govern the investigation. That being the case, it isn’t enough to know the laws, regulations, and policies of your company’s location, even if your company only resides in one geographical location. During an incident, it is also important to know and understand (or have the ability to learn about) the laws and regulations in the places that are directly or indirectly “involved” in the execution of the incident.
In the U.S., for example, many of the fifty states have their own laws on employee email privacy. Should it seem necessary to monitor an employee’s email during the investigation of an incident, most states require that the company have a written policy stating that the employee has no expectation of privacy on company-owned computers. Some states even require that the employee physically sign the policy statement, acknowledging awareness of the policy. To monitor an employee’s email without such a policy in a state that protects the employee’s right to privacy, even on company email, could be a criminal offense.
At the federal level, the U.S. has several blanket laws and guidance documents dealing with computer crime that an incident responder may find useful to know and keep handy -- particularly when trying to learn what your legal recourse may be for a given type of incident. We include references to several of these in Chapter 8.
The way to avoid a difficult and embarrassing situation is to be familiar with the laws in your area and the policies of the company. If the company’s policies are not sufficient to conduct incident response investigations adequately, consider petitioning the senior management to create (or approve) a policy on employee privacy. Clearly, the General Counsel needs to be involved in the drafting of such a policy, as it needs to be tied to the laws and regulations the company is required to adhere to.
The reason that we highlight “policy” as a common pitfall is that many teams become entrenched in drafting policy and not involved or focused enough on the operational issues of incident response. While it is true that any company needs adequate information protection policies, regardless of whether it chooses to adopt a formal incident response program, the policies should enable the team’s activities. Furthermore, it is useful for an incident response team to participate in the drafting, reviewing, or revising of the company’s information protection policies, but that should not be the team’s principal activity. Policies need to exist, but they should also be kept in perspective. An incident response team is a customer service organization, and policy is but one component of what constitutes effective incident response.
Few issues in incident response are as dicey as meeting customer expectations. When a customer is in crisis, there is a strong tendency to expect different things from the team than what the team is capable of providing. While the team must remain customer-service oriented, it must also be cognizant of the governing laws of physics, business, and reality that limit what it can do. Great care must be taken to listen to the customer’s requirements and fears. The team must then clearly articulate what it is capable of providing. If the team is not capable of providing what the customer wants, it is incumbent on the team leader to explain to the client why his needs can’t be met, and to present an alternative solution set or suggest alternatives. Once a solution set is agreed to, it is still incumbent on the team leader to keep the customer informed and continue to synchronize the customer’s expectations with the team’s plans and capabilities. This must be done throughout the process until resolution of the incident.
This sort of customer relationship is vital to the success of an incident response operation. Always be responsive.
Get Incident Response now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
