Building Event-Driven Microservices, 2nd Edition

Chapter 14. Streaming SQL

Streaming SQL is a declarative mechanism for building microservice-like queries. It’s a handy way to reduce the overhead on getting started with stream processing, and is accessible to those of you who are familiar with plain old SQL. You tell the SQL API what you want the outcome to look like, and it does all the work to implement it using the underlying engine.

Streaming SQL is most commonly implemented as a layer on top of a full-featured streaming framework, as introduced in “The Streaming SQL Query”. As examples, both Apache Flink and Apache Spark provide their own streaming SQL layers on top of their lower-level APIs. Flink provides Flink SQL, and Apache Spark provides Spark SQL (in conjunction with its structured streaming product). While there are other projects available out there that offer SQL APIs on top of streaming frameworks, this chapter focuses solely on these two open source Apache projects.

That being said, the principles of the streaming SQL remain largely the same regardless of the engine or framework you choose. The syntax and some of the lower-level details may differ, but the general idea remains the same: a SQL-syntax stream-processing API that lets you avoid the deeper abstraction layers of the streaming framework.

It’s important to point out that there is no definitive streaming SQL standard at the moment (ANSI or otherwise). It’s a bit of a wild west, with different frameworks offering differing syntaxes and semantics from ...