Mastering Kafka Streams and ksqlDB

Chapter 11. Intermediate and Advanced Stream Processing with ksqlDB

In the last chapter, we learned how to perform basic data preprocessing and transformation tasks using ksqlDB. The SQL statements we discussed were stateless, and they allowed us to filter data, flatten complex or nested structures, use projection to reshape data, and more. In this chapter, we will deepen our understanding of ksqlDB by discussing some data enrichment and aggregation use cases. Most of the statements we will discuss are stateful (e.g., involve multiple records, which is required with joins and aggregations) and time-based (e.g., windowed operations), making them more complex under the hood, but also more powerful.

Some of the topics we will cover in this chapter include:

• Using joins to combine and enrich data

• Performing aggregations

• Executing pull queries (i.e., point lookups) against materialized views using the CLI

• Working with built-in ksqlDB functions (scalar, aggregate, and table functions)

• Creating user-defined functions using Java

We will use the Netflix change-tracking tutorial from the previous chapter to introduce many of these concepts (see “Tutorial: Monitoring Changes at Netflix” if you need a recap). However, some topics, including ksqlDB functions, will be introduced as standalone discussions near the end of the chapter.

Let’s get started by reviewing the project setup steps.

Project Setup

If you would like to reference the code as we work our way through each topology ...