Anomaly Detection for Monitoring

Chapter 2. A Crash Course in Anomaly Detection

This isn’t a book about the overall breadth and depth of anomaly detection. It is specifically about applying anomaly detection to solve common problems that the DevOps community faces when trying to monitor the types of systems that we manage the most.

One of the implications is that this book is mostly about time series anomaly detection. It also means that we focus on widely used tools such as Graphite, JavaScript, R, and Python. There are several reasons for these choices, based on assumptions we’re making.

• We assume that our audience is largely like ourselves: developers, system administrators, database administrators, and DevOps practitioners using mostly open source tools.

• Neither of us has a doctorate in a field such as statistics or operations research, and we assume you don’t either.

• We assume that you are doing time series monitoring, much like we are.

As a result of these assumptions, this book is quite biased. It is all about anomaly detection on metrics, and we will not cover anomaly detection on configuration, comparing machines amongst each other, log analysis, clustering similar kinds of things together, or many other types of anomaly detection. We also focus on detecting anomalies as they happen, because that is usually what we are trying to do with our monitoring systems.

A Real Example of Anomaly Detection

Around the year 2008, Evan Miller published a paper describing real-time anomaly detection in operation ...