Data Lake for Enterprises

Core architecture principles of Kafka

The main motivation behind Kafka when developed by LinkedIn’s engineering team was

To create a unified messaging platform to cater to real-time data from various applications in a big organization.

- LinkedIn

There are core architecture principles based on which Kafka was conceived and designed. The bulleted points sum up these principles:

• Maximize performance (compression and B-tree usage is an example)
• Wherever possible, core kernel capabilities to offload work to drive optimization and performance (zero-copy and direct use of Linux filesystem cache is an example)
• Distributed architecture
• Fault tolerance
• Durability of messages
• Wherever possible, eliminate redundant work
• Offload responsibility of tasks ...