Mastering Kafka Streams and ksqlDB

Chapter 7. Processor API

Just a few chapters ago, we embarked on our journey to learn about Kafka Streams. We started with Kafka Streams’ high-level DSL, which allows us to build stream processing applications using a functional and fluent interface. This involves composing and chaining stream processing functions using the library’s built-in operators (e.g., filter, flatMap, groupBy, etc.) and abstractions (KStream, KTable, GlobalKTable).

In this chapter, we will explore a lower-level API that is available in Kafka Streams: the Processor API (sometimes called PAPI). The Processor API has fewer abstractions than the high-level DSL and uses an imperative style of programming. While the code is generally more verbose, it is also more powerful, giving us fine-grained control over the following: how data flows through our topologies, how stream processors relate to each other, how state is created and maintained, and even the timing of certain operations.

Some of the questions we will answer in this chapter include:

• When should you use the Processor API?

• How do you add source, sink, and stream processors using the Processor API?

• How can you schedule periodic functions?

• Is it possible to mix the Processor API with the higher-level DSL?

• What is the difference between processors and transformers?

As usual, we will demonstrate the fundamentals of the API through a tutorial, answering the preceding questions along the way. However, before we show you how to use the Processor ...