Mastering Kafka Streams and ksqlDB

Chapter 10. Stream Processing Basics with ksqlDB

In this chapter, we will learn how to use ksqlDB to perform some common stream processing tasks. The topics we will explore include:

• Creating streams and tables

• Leveraging ksqlDB data types

• Filtering data using simple Boolean conditions, wildcards, and range filters

• Reshaping data by flattening complex or nested structures

• Using projection to select a subset of the available fields

• Using conditional expressions to handle NULL values

• Creating derived streams and tables, and writing the results back to Kafka

By the end of this chapter, you will be well prepared to handle basic data pre-processing and transformation tasks using ksqlDB’s SQL dialect. Furthermore, while the SQL constructs we explored in the preceding chapter leveraged ksqlDB’s Kafka Connect integration under the hood, all of the SQL statements we will use in this chapter will leverage ksqlDB’s Kafka Streams integration. This is where we will really start to see how powerful ksqlDB is, since with just a few lines of SQL, we will be able to build full-blown stream processing applications. As usual, we will explore this chapter’s topics by building an application that leverages the related language constructs. So without further ado, let’s introduce our tutorial.

Tutorial: Monitoring Changes at Netflix

Netflix invests billions of dollars in video content every year. With so many film and television shows in production at once, communicating updates to various ...