Kafka acts as a write-ahead log that records messages to a persistent store and allows subscribers to read and apply these changes to their own stores in a system-appropriate timeframe. Common subscribers include live services that do message aggregation or other processing of these streams, as well as Hadoop and data warehousing pipelines, which load virtually all feeds for batch-oriented processing. Overall, Kafka was built with the following goals in mind:
- Loose coupling between message producers and message consumers
- Persistence of message data for different consumers and failure handling
- Maximize end-to-end throughput with low latency components
- Managing diverse data formats and types
- Scaling servers linearly without affecting ...
