Mastering Kafka Streams and ksqlDB

Chapter 4. Stateful Processing

In the previous chapter, we learned how to perform stateless transformations of record streams using the KStream abstraction and a rich set of stateless operators that are available in Kafka Streams. Since stateless transformations don’t require any memory of previously seen events, they are easy to reason about and use. We treat every event as an immutable fact and process it independently of other events.

However, Kafka Streams also gives us the ability to capture and remember information about the events we consume. The captured information, or state, allows us to perform more advanced stream processing operations, including joining and aggregating data. In this chapter, we will explore stateful stream processing in detail. Some of the topics we will cover include:

• The benefits of stateful stream processing

• The differences between facts and behaviors

• What kinds of stateful operators are available in Kafka Streams

• How state is captured and queried in Kafka Streams

• How the KTable abstraction can be used to represent local, partitioned state

• How the GlobalKTable abstraction can be used to represent global, replicated state

• How to perform stateful operations, including joining and aggregating data

• How to use interactive queries to expose state

As with the previous chapter, we will explore these concepts using a tutorial-based approach. This chapter’s tutorial is inspired by the video game industry, and we’ll be building a real-time ...