Anomaly Detection for Monitoring

Chapter 3. Modeling and Predicting

Anomaly detection is based on predictions derived from models. In simple terms, a model is a way to express your previous knowledge about a system and how you expect it to work. A model can be as simple as a single mathematical equation.

Models are convenient because they give us a way to describe a potentially complicated process or system. In some cases, models directly describe processes that govern a system’s behavior. For example, VividCortex’s Adaptive Fault Detection algorithm uses Little’s law¹ because we know that the systems we monitor obey this law. On the other hand, you may have a process whose mechanisms and governing principles aren’t evident, and as a result doesn’t have a clearly defined model. In these cases you can try to fit a model to the observed system behavior as best you can.

Why is modeling so important? With anomaly detection, you’re interested in finding what is unusual, but first you have to know what to expect. This means you have to make a prediction. Even if it’s implicit and unstated, this prediction process requires a model. Then you can compare the observed behavior to the model’s prediction.

Almost all online time series anomaly detection works by comparing the current value to a prediction based on previous values. Online means you’re doing anomaly detection as you see each new value appear, and online anomaly detection is a major focus of this book because it’s the only way to find system problems as they ...