Building Data Streaming Applications with Apache Kafka

Additional producer configuration

There are other optional configuration properties available for Kafka producer that can play an important role in performance, memory, reliability, and so on:

• buffer.memory: This is the amount of memory that producer can use to buffer a message that is waiting to be sent to the Kafka server. In simple terms, it is the total memory that is available to the Java producer to collect unsent messages. When this limit is reached, the producer will block the messages for max.block.ms before raising an exception. If your batch size is more, allocate more memory to the producer buffer.

Additionally, to avoid keeping records queued indefinitely, you can set a timeout using request.timeout.ms. If this timeout expires ...