Data Lake for Enterprises

Scale-out architecture with Kafka

Main principles on which Kafka works have been covered in this chapter earlier. We won't cover those again here; however below are the main reasons for scale-out architecture in Kafka:

• Partition: Splits a topic into multiple partitions and increasing partitions is a mechanism of scaling.
• Distribution: Cluster can have one or more brokers and these brokers can be increased to achieve scaling.
• Replication: Similar to partitions, multiple replication of a message is there for fault-tolerance and this aspect also brings in scalability in Kafka.
• Scaling: Each consumer reads a message from a single partition (of a topic) and to scale out we add more consumers and the newly added consumers read the message from ...