Building Data Streaming Applications with Apache Kafka

Data and state persistence

Data integrity, safety, and availability are some of the key requirements of any successful streaming application solution. If you give these factors a thought, you will understand that to ensure integrity, safety, and availability, persistence plays an important role. For example, it is absolutely essential for any streaming solution to persists its state. We often call it checkpointing. Checkpointing enables streaming applications to persist their states over a period of time and ensures recovery in case of failures. State persistence also ensures strong consistency, which is essential for data correctness and exactly-once message delivery semantics.

Now you must have understood why persisting state is important. ...