Fast Data Architectures for Streaming Applications, 2nd Edition

Chapter 4. How Do You Analyze Infinite Data Sets?

Infinite data sets raise important questions about how to do certain operations when you don’t have all the data and never will. In particular, what do classic SQL operations like GROUP BY and JOIN mean in this context? What about statistics like min, max, and average?

A theory of streaming semantics has emerged that provides the answer. Central to this theory is the idea that aggregation operations like these make sense only in the context of windows of data, often over a fixed range of time.

Apache Beam, an open source streaming engine based on Google Dataflow, is arguably the streaming engine with the most sophisticated formulation of these semantics. It has become the gold standard for defining how precise analytics should be performed in real-world streaming scenarios. Although Beam is not as widely used as the other streaming engines discussed in this report, the designs of these engines were strongly influenced by Beam.

To actually use Beam, a third-party “runner” is required to execute Beam data flows. In the open source world, this functionality has been implemented for Flink and Spark, while Google’s own runner is its cloud service, Cloud Dataflow. This means you can write Beam data flows and run them with these other tools. Not all constructs defined by Beam are supported by all runners. The Beam documentation has a Capability Matrix that shows what features each runner supports. There are even semantics defined that ...