Kafka architecture components

Let's take a look at each component in detail:

  • Producers: Producers publish messages to a specific Kafka topic. Producers may attach a key to each message record. By default, producers publish messages to topic partitions in round robin fashion. Sometimes, producers can be configured to write messages to a particular topic partition based on the hash value of message key.
  • Topic: All messages are stored in a topic. A topic is a category or feed name to which records are published. A topic can be compared to a table in a relational database. Multiple consumers can be subscribed to a single topic to consume message records.
  • Partition: A topic is divided into multiple partitions. Kafka offers topic parallelism by ...

Get Modern Big Data Processing with Hadoop now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.