Building Data Streaming Applications with Apache Kafka

Memory

Kafka is highly dependent on the file system for storing and caching messages. All the data is written to the page cache in the form of log files, which are flushed to disk later. Generally, most of the modern Linux OS use free memory for disk cache. Kafka ends up utilizing 25-30 GB of page cache for 32 GB memory.

Moreover, as Kafka utilizes heap memory very efficiently, 4-5 GB of heap size is enough for it. While calculating memory, one aspect you need to remember is that Kafka buffers messages for active producers and consumers. The disk buffer cache lives in your RAM. This means that you need sufficient RAM to store a certain time of messages in cache. While calculating buffer requirements, there are two things you should keep in ...