Building Real-Time Analytics Systems

Chapter 2. The Real-Time Analytics Ecosystem

In the previous chapter, we discussed the fundamentals of real-time analytics: the events, streams, and different types of real-time analytics systems. Real-time analytics is a vast domain consisting of multiple tools, technologies, and practices. Before building real-time analytics applications, it will be crucial for you to identify the technology ecosystem around real-time analytics. The related tools and technologies work together to derive insights from streaming data.

In this chapter, we go back in time and discuss the first-generation real-time analytics systems, their early days, the influence of the Lambda architecture, and the challenges they faced. Then we discuss the modern real-time analytics ecosystem (or the modern streaming stack), its composition, and its outlook for the future.

We hope that by the end of this chapter, you will have a map that you can use to analyze any future data infrastructure products and understand where they fit into the real-time analytics application architecture.

Defining the Real-Time Analytics Ecosystem

Imagine you are building an ecommerce web application and have to play multiple roles as architect, backend developer, and operator. As an architect, you will decide on factors like cloud provider and geographical region, container orchestration mechanism, and high-availability requirements. As a backend developer, you will not only write the application code but also work closely with other ...