Incident Response Primer

Chapter 1. Incident Response Basics

Consulting whatever source you use these days for your news or information intake will, as often as not it seems, include word of the latest attack on some business or other organization. As of this writing, many municipal governments have been recently hit with ransomware attacks. Health care entities have been highly susceptible, too. Pick an industry and someone out there is trying to take advantage of it in some fashion or another. Because of the high prevalence of attackers looking to either make a statement, make a buck, or gather information, incident response has become one of the most important sectors of the tech industry.

Responding to incidents is not exactly a new field, but with so many attackers out there, incident response capability is in greater demand than ever before. What you may not be aware of is the increasing numbers of adversarial groups that have sprung up, groups looking to obtain illegal access to systems, networks, and information. (There are many reasons why these groups emerge, which will be covered later on.) Inexpensive computing power and network bandwidth have made it much easier for criminals to operate. Essentially, the whole criminal operation has become democratized, meaning it’s easy for anyone to get into the game, very often without any penalties. You rolls the dice, you takes your chances that you’ll score big. And there is certainly plenty of score to be had.

When it comes to incident response, though, there is much that is misunderstood. This makes it hard for organizations to do the right thing. One problem is nomenclature; people throw words around related to information security and, very often, those words are used incorrectly. This makes it difficult to establish a normalized understanding of what is happening, how it is happening, and what to do about it.

This brings me to a pet peeve; for the last several years, the words hack and hacked have been used a lot, to the point where they have become essentially meaningless. When preteens get their friend’s phone and post to Facebook that they “hacked” the phone, that is not an accurate meaning of this word. When these words are used in news stories, it’s hard to understand what has actually happened. Personally, I find it much easier when describing such incidents to use clear language that identifies the reality of the situation: these are not hackers we are generally talking about who do these things, these are criminals. They are thieves. They are stealing data that belongs to someone else. Let’s call them what they are. When someone robs a jewelry store or a bank, we don’t use ambiguous terms to talk about them. We call those people thieves and criminals. When it comes to hunting these thieves and criminals, they become adversaries or attackers. Simple, clear language that doesn’t obscure what is actually happening.

Other terms are often misused and misunderstood. One of the most significant of these terms is risk, which is at the core of information security and as incident response is a function of information security, it’s relevant here. When we are looking to respond to an incident, there are other terms it’s helpful to understand. Those are events and incidents. Finally, there is threat. This is often also not well understood. Talking about different threats can be helpful to explain not only what a threat is, but also to describe what is possible to expect from threats.

Risk

How often have you heard someone say that there is a risk that something will happen? Usually, there is something at least vaguely negative about the thing that may occur. You may hear someone say that there is a risk that it will rain today. What they really mean is that there is a chance it will rain. Because they don’t really want it to rain, they misuse the word “risk” rather than just using the clear, simple word “chance.” When we use the word “chance,” we mean there is a measurable probability of the occurrence of the thing we’re talking about. Although probability is a function of risk, it is not the only function or aspect of risk.

Humans tend to catastrophize. Sometimes when you hear someone talking about risk, you hear them talking about really bad things. If you ask someone about a risk they may be exposed to, usually you will hear them respond with something that could have the worst possible outcome. The risk they are exposed to is a meteor striking the Earth and killing all human inhabitants. It sounds silly, doesn’t it? What they are talking about is the worst loss they can conceive of. While loss is a function or aspect of risk, it is not the only one.

Measuring risk means you need to be able to measure the probability and the loss. Risk is a function of both probability and loss. What I have often heard people talk about when asked about personal risk is driving. They perceive there is great risk when they are driving. Much of this perception comes from uncertainty. They feel they are great drivers (who doesn’t think they are a great driver, after all?), but it’s the other person. Someone else could be on the phone and speed through a red light, causing death or a loss of limb or sight or some other catastrophe. Uncertainty is not a factor of risk. If you aren’t in control of something happening, that doesn’t mean there is more risk. It just means there are factors you have no control over.

A rational look at risk requires not succumbing to fear, uncertainty, and doubt. It requires a clear-headed look at not only the potential loss, but also the probability of that loss. Think about the driving example. How many hours a year are you on the road? Out of all those hours, how many accidents were you in within the last 12 months? Now, extend that beyond 12 months. How many hours in the last 2 years? 5 years? How many accidents? Looking at that, what is the probability of you getting into an accident in the future? Beyond that, if you have been in an accident, what was the outcome? A dented fender? Surely, this is a loss that is significantly less than someone being maimed. These are the types of accidents that are most likely to happen.

When it comes to information security, you apply the same sort of logic. You have to be able to measure the probability of something happening. Then, consider the loss you would incur from that something happening. When you have a probability and a loss, you have a risk. You can apply calculations to this sort of risk. Just saying ZOMG!!! EVERYTHINGZ BAD!! isn’t every helpful. The sky is not falling, and even if it were, would running around screaming be very helpful? Being able to make informed decisions can help you prepare for when bad things, inevitably, happen.

Events

An event is an observable occurrence. Anything could be an event. Any change in state to a system is an event. When someone logs in to a system, that’s an event. When a network interface goes down, say from the cable being unplugged, that’s an event. Events can be utterly meaningless. They are simply observable changes in a system. So, going forward, we can talk about events without anyone getting too upset or stressed. An event is just something that happens, which could mean absolutely nothing at all.

As an individual, you create dozens or even hundreds of events on just your system every day. If you look at whatever event log you have available to you, you will see a large number of entries. If you are on a Windows system, you can see the events in the Event Viewer application. On macOS, they are in the console, which you can see in Figure 1-1. On Linux, syslog keeps track of events, usually in files in the /var/log directory. Events are constantly being added to these logfiles. If you were to take that out to an enterprise scale, there are hundreds of thousands or millions of events happening, maybe every hour, depending on the size of the organization.

Figure 1-1. Console in macOS

Just because there are a large number of events happening every minute, hour, or day, it doesn’t mean you should ignore them. Events are not always benign. I could stub my toe in the night. It’s an event because it’s an observable change from normal (walking across the floor unimpeded would be normal for me, ideally). However, it could also be something else in addition to being an event, depending on how badly damaged my toe got in the process.

Incidents

An incident is an event with an adverse outcome. It is still an event, though. The reason it is both an event and an incident is that any incident is an observable change from the normal state. If I stubbed my toe and had to go to the emergency room (my dogs are of no use at all when it comes to bandaging wounds), that would be an incident because it’s an event with an adverse outcome. This, however, is a fairly benign incident in the grand scheme of things. On top of that, I wouldn’t convene an incident response team to address my stubbed toe.

A login to any system is an event. It’s an event because before the user logged in, the state of the system or account was not logged in. When the login occurred, there was a change, which made it an event. Follow so far? A login is an event. However, if we have two simultaneous logins with the same user credentials, it’s two events. Still, maybe nothing at all to be concerned about, because users log in. If one of the logins were to be happening from China, while the other was taking place in Denver, we probably have an incident on our hands. The geographic disparity is problematic since only one person should ever be in possession of an account and it would be hard for that single person to be in both China and Denver at the same time.

An incident doesn’t necessarily mean we jump straight to all-out incident response mode, though. It should mean that some investigation is in order, but that may be a simple investigation with an easy outcome—terminate the spurious login from China (or Denver, depending on where the user is normally located), disable the account, and make sure the user changes their password. This could be simple. However, it isn’t necessarily simple. This is why we have incident response and incident response teams. We’re talking about what incident response means here, and an incident response team is the collection of people who are responsible for managing the response.

Threats

You could also be wondering where threats fall into this whole boiling pot of words and concepts. A threat is anything that could potentially cause an adverse outcome. Threats come in all sorts of shapes and sizes. Dave in your data center could be a threat because Dave doesn’t pay attention to where he is walking, with his face planted firmly in his phone, staring at his Twitter feed. Dave regularly trips over things in front of him. If he were to trip on or kick a power cable, it would cause an outage. The cause of that outage would be Dave. He is the threat. Don’t be like Dave. Don’t be the threat.

Today, there are a wide variety of threats in the world that are far more problematic than Dave. That is, unless Dave works for a company that relies on ecommerce and Dave keeps kicking the power out of the load balancers, meaning the business gets no revenue until someone figures out the power is out and gets the load balancer back online so customers can be served. Let’s assume Dave isn’t quite that bad. Some of the threats you can expect to find today are as follows:

Hacktivists

A hacktivist is someone with an axe to grind. They have something to say and they have resorted to computers and networks to make their case. These people are often nothing more than a nuisance. Sometimes they are a giant nuisance. Middle Eastern hacktivists sometimes don’t understand Western culture and so they may target an entirely innocent bystander. This was the case several years ago when a Middle Eastern hacktivist group set out to protest the content of a YouTube video, so they launched ongoing, massive denial of service attacks against several prominent US banks. Though there was no disruption of service to customers, because the banks were able to withstand the sustained 600+-megabit-per-second onslaughts, it was more than simply a nuisance.

Criminals

These are groups who are looking for money, almost any way they can get it. Phishing attacks are often from criminals trying to either gain information they can later monetize (sell) or just trying to get money directly from the victim—this may be ransomware, for instance, where the victim’s data is encrypted until they pay money. You can see a sample phishing email in Figure 1-2. This is probably an attempt to collect information from me, which may include my Amazon login. The criminal would take my Amazon login and order products they could then resell somewhere. And I’d be stuck with the bill for it all. Don’t be like Dave. Don’t click the link.

Figure 1-2. Phishing email

Nation-state actors

A nation-state actor is any individual or group that has sponsorship from a country, whether it’s the government proper or the military. Nation-state actors (an actor is the person or group actually engaged in the activity) may have different motivations. Criminals are looking for money. Nation-state actors may also be looking for money, as in the case of North Korea (which we discuss later in this report). Nation-state actors may also be looking for information, as in the case of China, which has historically sought intellectual property to allow its state-run businesses to compete on a global basis.

You will also hear the term advanced persistent threat (APT). An APT is any group that utilizes techniques that allow them to gain access to an environment and remain in that environment. Persistence means they remain in the environment or system even after systems have been rebooted or after the attacker has disconnected. When the attacker gets access, they use multiple means to avoid being detected and be able to continue to have access to systems and networks over a long period of time. Often, if it were left up to the attacker anyway, the access is over the course of years rather than hours or days.

One thing to keep in mind when it comes to most of these threats is that the actors behind them are motivated. It could be financial motivation or the motivation to act in the best interest of their country. You should think of these adversaries as businesses. They are in business to make money or gather intelligence. They are staffed appropriate to their needs, and quite often, they use shifts of workers to gather information or continue to get deeper access to other systems in the environment, always looking for whatever they are in business for—money, information, access, etc. Because these groups are motivated, if they have targeted you, it’s because you have something they want. Kicking them out won’t change that motivation. They will keep trying to get access again because you continue to have something they want.

Sometimes, what they want, or what they will accept, is just access to systems. You may have heard the term botnet. This is a collection of systems that have malicious software on them, allowing attackers to control those systems remotely. Those botnet systems (bots) can be used in denial of service attacks, like the ones mentioned against US banks. They can also be used as cheap computing power. Businesses can be started to sell cheap pharmaceuticals, for example. They use your systems for their web servers and also their ecommerce servers, taking orders and processing payments.

There are a lot of threats in the world today. You can categorize them easily enough, but the question is whether you are prepared to respond when those threats come after your business. This is what we can start to address in the next chapter.