Mastering Hadoop 3

Apache Kafka architecture

All messages in the Kafka topics are a collection of bytes which is represented as an array. The Producers are the main applications which stores information in Kafka queues and sends messages to Kafka topics that stores all types of messages. All the topics are further categorised into partitions in which each partition stores messages in the sequence that they arrive. Basically, there are two major operations that Producers and Consumers can perform in Kafka, the Producers append to the end of the write-ahead log files and Consumers fetch messages from these log files belonging to a given topic partition. Physically, each topic is spread over different Kafka brokers, which host one or two partitions of each topic. ...