Building Data Streaming Applications with Apache Kafka

Producer read and consumer write choices

We talked about choices of hardware one should make in case of high volumes. In this section, we will talk about some of the important techniques to manage high throughput and low latency in case of reading and writing high volume data in Kafka.

We are listing some of the techniques that you can use while writing or reading data:

• Message compression: The producer generates the compression type of all the data. The value for the compression type property are none, GZIP, Snappy, or lZ4. More batching leads to better compression ratio because compression happens on the entire batch. You may need to compromise with more CPU cycles to complete the compression process, but it will definitely save the network ...