Data Lake for Enterprises

Persistence of data in Kafka using topics

High-level abstraction provided by Kafka where the data pushed by producers is persisted, having a defined name. To cater to replication and fault tolerance each topic is divided into multiple partitions and each partition is distributed in different brokers inside the Kafka cluster.

The topic doesn't keep a note of consumers who have consumed the messages inside the topic, rather the full control of monitoring is offloaded to the consumer and it's their responsibility to keep the track of messages consumed. The consumer comes to a topic and collects a message beginning from a particular message ID and this has to be managed by the consumer.

The following figure shows the inner structure of a topic, ...