Streaming Databases

Chapter 3. Serving Real-Time Data

In Chapter 2, we had the stream processing platform transform the data and place it into a sink topic. The preprocessed data is now residing in a topic in the streaming platform. In Figure 3-1, the sink topic and OLAP data store are highlighted in the analytical plane.

The next thing we need to do is to serve real-time data to the consumers. In this chapter, we’ll talk about delivering enriched real-time data to the end user. This stage of the real-time data pipeline is the last mile streaming data takes before it’s presented to the end user.

Figure 3-1. The preprocessed data is available in the sink topic for real-time analytical serving to user-facing dashboards or applications

Real-Time Expectations

To serve real-time analytics to the consumers we’ve identified (humans and applications), a set of service-level agreements (SLAs) should be considered. In our clickstream use case, we didn’t specify requirements for the end user or application. However, since we want to serve analytics in real time, we should consider some metrics:

Latency

Measures the time it takes for an analytics query or computation to complete and return results. In real-time analytics, low latency is crucial to provide near-instantaneous insights to users. SLA metrics may define acceptable latency thresholds, such as average response time or maximum response time, to ...