Learning Apache Apex

Incremental state saving

Operators can hold a large amount of state in memory to facilitate fast access for low-latency processing. At the same time, this state also needs to be recoverable in the event of failure. With the growing state, the time to checkpoint increases, leading to larger pauses in processing, and eventually, certain processing time thresholds are exceeded, to the crash of the operator. To overcome this and achieve the twin goals of fault tolerance and fast processing, state needs be saved in an efficient manner, such that the size of the checkpointed state is capped.

For this purpose, the Apex library has Managed State, a fault-tolerant, scalable bucketing mechanism that uses files to persist the data (the Hadoop distributed ...