Fast Data: Smart and at Scale

Chapter 3. Recipe: Integrate Streaming Aggregations and Transactions

Idea in Brief

Increasing numbers of high-speed transactional applications are being built: operational applications that transact against a stream of incoming events for use cases like real-time authorization, billing, usage, operational tuning, and intelligent alerting. Writing these applications requires combining real-time analytics with transaction processing.

Transactions in these applications require real-time analytics as inputs. Recalculating analytics from base data for each event in a high-velocity feed is impractical. To scale, maintain streaming aggregations that can be read cheaply in the transaction path. Unlike periodic batch operations, streaming aggregations maintain consistent, up-to-date, and accurate analytics needed in the transaction path.

This pattern trades ad hoc analytics capability for high-speed access to analytic outputs that are known to be needed by an application. This trade-off is necessary when calculating an analytic result from base data for each transaction is infeasible.

Let’s consider a few example applications to illustrate the concept.

Pattern: Reject Requests Past a Threshold

Consider a high-request-volume API that must implement sophisticated usage metrics for groups of users and individual users on a per-operation basis. Metrics are used for multiple purposes: they are used to derive usage-based billing charges, and they are used to enforce a contracted quality of ...