Apache Kafka is a high-throughput, distributed, fault tolerant, and replicated messaging system that was first developed at LinkedIn. The use cases of Kafka vary from log aggregation to stream processing to replacing other messaging systems.
Kafka has emerged as one of the important components of real-time processing pipelines in combination with Storm. Kafka can act as a buffer or feeder for messages that need to be processed by Storm. Kafka can also be used as the output sink for results emitted from the Storm topologies.
In this chapter, we will cover the following topics: