Machine Learning and Security

Chapter 3. Anomaly Detection

This chapter is about detecting unexpected events, or anomalies, in systems. In the context of network and host security, anomaly detection refers to identifying unexpected intruders or breaches. On average it takes tens of days for a system breach to be detected. After an attacker gains entry, however, the damage is usually done in a few days or less. Whether the nature of the attack is data exfiltration, extortion through ransomware, adware, or advanced persistent threats (APTs), it is clear that time is not on the defender’s side.

The importance of anomaly detection is not confined to the context of security. In a more general context, anomaly detection is any method for finding events that don’t conform to an expectation. For instances in which system reliability is of critical importance, you can use anomaly detection to identify early signs of system failure, triggering early or preventive investigations by operators. For example, if the power company can find anomalies in the electrical power grid and remedy them, it can potentially avoid expensive damage that occurs when a power surge causes outages in other system components. Another important application of anomaly detection is in the field of fraud detection. Fraud in the financial industry can often be fished out of a vast pool of legitimate transactions by studying patterns of normal events and detecting when deviations occur.

Terminology

Throughout the course of this chapter, we use ...