Data Lake for Enterprises

Setting up multi-broker cluster

While we may be setting up the Kafka server on a single node deployment in our example code, we can always have multiple broker deployment for single node level resiliency. In that case, if one broker fails, the other broker is still available to serve the messages.

Setting up multiple brokers on a single node is very straightforward and involves changing the following settings in ${KAFKA_HOME}/config/server.properties.

Configuration Parameter	Description
broker.id	This should always be unique in a Kafka cluster for each of the broker instances
port	The port must be different if multiple brokers are to be set up on the same node
logs.dir	The log location for each of the brokers must be defined on ...