Designing Fast Data Application Architectures

Chapter 6. Serving

Fast Data applications are built to deliver continuous results and are consumed by other apps and microservices. Some examples include real-time dashboards for monitoring business Key Performance Indicators (KPIs), or an application to enrich the analytical capabilities of Business Intelligence (BI) software, or an aggregation of messages to be queried by a RESTful API. Applications may also apply machine learning (ML) techniques to the data, such as scoring an ML model, or even train a model on the fly.

Let’s explore some patterns we can use in a serving layer. We can use a Big Table–based technology, such as Cassandra or HBase (or, more traditionally, RDBMS), that is continuously updated. Users then consume the data with client applications that read from them.

Importing batch data into highly indexed and aggregated data stores used with analytical data-mining BI tools is a common practice. A newer trend is to use Streaming SQL to apply analytical transformations on live data. Streaming SQL is supported by all major stream processors including Apache Spark, Apache Flink, and Kafka Streams.

Finally, it is possible to serve data directly from the message backbone. For example, we can consume messages from a Kafka topic into a dashboard to build a dynamic, low-latency web application.

Sharing Stateful Streaming State

When running stateful streaming applications, another possibility is to share a view of that state directly. This is a relatively new ability in ...