Building Data Streaming Applications with Apache Kafka

At least once delivery 

In the producer context, at least once delivery can happen if acks are lost in network translation. Suppose the producer has configuration of acks=all. This means the producers will wait for success or failure acknowledgement from the brokers after messages are written and replicated to relevant brokers.

In case of timeout or some other kind of error, the producer re-sends those messages assuming that they are not written to topic partitions. But what if the failure happens right after the messages are written to Kafka topic but ack can not be sent? In that case, the producer will retry sending that message, resulting in messages being written more than once.

In this kind of scenario, generally, message de-duplication ...