Kafka in Action

8 Kafka storage

This chapters covers

• How long to retain data
• Data movement into and out of Kafka
• Data architectures Kafka enables
• Storage for cloud instances and containers

So far we have thought of our data as moving into and out of Kafka for brief periods of time. Another decision to consider is where our data should live long term. When you use databases like MySQL or MongoDB^®, you may not always think about if or how that data expires. Rather, you know that the data is (likely) going to exist for the majority of your application’s entire lifetime. In comparison, Kafka’s storage logically sits somewhere between the long-term storage solutions of a database and the transient storage of a message broker, especially if we think of message brokers ...