Learning Apache Apex

Kafka input

The Kafka input operator consumes messages from the partitions of a Kafka topic and emits the message bytes to downstream operators for further processing (parsing, filtering, and so on). While this is simple functionality, let's see how aspects like scalability, efficient resource utilization, and fault tolerance are addressed.

As explained, Kafka scales with partitions; so, the operator needs the ability to automatically scale with the partitioning of the Kafka topic and the resources required for the scaling need to be allocated dynamically. The following diagram shows two different partition mapping strategies that are supported out of the box:

On the left-hand side, we see each Kafka partition being consumed by one partition ...