Building Event-Driven Microservices, 2nd Edition

Chapter 9. Deterministic Stream Processing

Event-driven microservices typically have topologies that are more complex than those introduced in the previous chapter (see “Microservice Topology”). They consume events from multiple partitioned event streams, maintain state, repartition data, and emit new events to other streams. They are also subject to all the same failure modes and faults as any other application, which can result in out-of-order events, late events, and questions around how events will be processed in actuality.

Here are the three main questions addressed in this chapter:

• How does a microservice choose the order of events to process when consuming from multiple partitions?

• How does a microservice handle out-of-order and late-arriving events?

• How do we ensure that our microservices produce deterministic results when processing streams in near–real time versus when processing from the beginning of the streams?

We can answer these questions by examining timestamps, event scheduling, watermarks, and stream times, and how they contribute to deterministic processing. Bugs, errors, and changes in business logic will also necessitate reprocessing, making deterministic results important. This chapter also explores how out-of-order and late-arriving events can occur, strategies for handling them, and mitigating their impact on our workflows.

Note

This chapter is fairly information-dense despite my best efforts to find a simple and concise way to explain the key ...