Real-Time Analytics: Techniques to Analyze and Visualize Streaming Data

Chapter 11Beyond Aggregation

For many organizations, aggregation and visualization are the end of the road. Dashboards are created, aggregates are graphed, and reports are generated. To what end? The answer is usually so that “decision-makers” can take the “pulse” of whatever system is being monitored. Interpreting this statement a bit, it would seem to imply that the role of these systems is to surface information to a human decision-maker so that they can take some action that affects the system in a desirable way or react to an undesirable change in that system.

But, these systems are all operating in real time and humans are not real-time creatures. We eat, sleep, and generally do things other than stare at the continuously updating dashboard. How do we keep up? For mission-critical things, the solution has usually been to have a large number of humans working in shifts to keep an eye on the systems.

This works fairly well for relatively small systems, but even reasonably sized systems like power plants quickly reach the limits of feasibility for the “herd of humans” approach. This leads to the introduction of automated elements of the real-time system, such as alarms and automatic shutdown.

These automated systems are the focus of this chapter—helping the human decision-maker do their job by allowing them to focus on anomalous behaviors or automating the moment-to-moment decision-making process altogether. This turns out to be easier said than done, but a number of approaches ...