Fast Data Architectures for Streaming Applications, 2nd Edition

Chapter 6. Example Application

Let’s finish with a look at an example application, similar to systems that several Lightbend customers have implemented.¹ Here, telemetry for IoT (Internet of Things) devices is ingested into a central data center. Machine learning models are trained and served to detect anomalies, indicating that a hardware or software problem may be developing. If any are found, preemptive action is taken to avoid loss of service from the device.

Vendors of networking, storage, and medical devices often provide this service, for example.

Figure 6-1 sketches a fast data architecture implementing this system, adapted from Figure 2-1, with a few simplifications for clarity. As before, the numbers identify the diagram areas for the discussion that follows. The bidirectional arrows have two numbers, to discuss each direction separately.

Figure 6-1. IoT anomaly detection example

There are three main segments of this diagram. After the telemetry is ingested (label 1), the first segment is for model training with periodic updates (labels 2 and 3), with access to persistent stores for saving models and reading historical data (labels 4 and 5). The second segment is for model serving—that is, scoring the telemetry with the latest model to detect potential anomalies (labels 6 and 7)—and the last segment is for handling detected anomalies (labels 8 and 9).

Let’s look at ...