Streaming Data for Real-Time AI Applications

Chapter 2. Real-Time Data as Part of Application Architecture

Modern applications operate in dynamic data ecosystems, requiring real-time processing and action. Traditional synchronous models struggle with growing data and latency demands, leading to the rise of real-time data pipelines. These pipelines enable asynchronous communication, high-throughput event handling, and consistency across distributed systems. Key architectures include microservices, modular monoliths, and serverless. Each uses real-time data effectively, but with distinct trade-offs.

Microservices Architecture

Microservices architecture is a design paradigm that breaks an application into a collection of small, independent services, each responsible for a specific function.

Services in a microservices suite have traditionally communicated over a network using lightweight protocols like HTTP or gRPC. This kind of communication has relied on synchronous request-response mechanisms with direct service-to-service calls, but it can create bottlenecks, thereby increasing complexity and reducing resilience. To address these challenges, many organizations turn to message brokers to communicate asynchronously through messaging and event streams rather than direct API calls. With event-driven messaging, services operate autonomously, processing events as they arrive. The backbone of streaming pipelines is built on event streaming platforms like Apache Kafka, Redpanda, Apache Pulsar, Google Pub/Sub, and solutions from ...