Learning Apache Apex

Kafka output

Compared to the Kafka input operator, the Kafka output operator implementation is rather simple. The output operator takes the tuple from the input port and hands it to the Kafka producer (part of the operator configuration is to specify the appropriate message serializer). However, things get more complicated when taking fault tolerance into account. In versions prior to 0.11, Kafka did not support transactions, and therefore, the onus was on the client to make sure messages are not lost or duplicated in the event of a failure and when exactly-once output is required.

For this purpose, the library has a specialized KafkaExactlyOnceOutputOperator. This operator is actually both a producer and consumer, because it needs to remember ...