Setting up multi-broker cluster

While we may be setting up the Kafka server on a single node deployment in our example code, we can always have multiple broker deployment for single node level resiliency. In that case, if one broker fails, the other broker is still available to serve the messages.

Setting up multiple brokers on a single node is very straightforward and involves changing the following settings in ${KAFKA_HOME}/config/server.properties.

Configuration Parameter Description
broker.id This should always be unique in a Kafka cluster for each of the broker instances
port The port must be different if multiple brokers are to be set up on the same node
logs.dir The log location for each of the brokers must be defined on ...

Get Data Lake for Enterprises now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.