Building Event-Driven Microservices

Chapter 6. Deterministic Stream Processing

Event-driven microservices usually have topologies that are more complex than those introduced in the previous chapter. Events are consumed and processed from multiple event streams, while stateful processing (covered in the next chapter) is required to solve many business problems. Microservices are also subject to the same faults and crashes as nonmicroservice systems. It is not uncommon to have a mixture of microservices processing events in near–real time while other, newly started microservices are catching up by processing historical data.

Here are the three main questions addressed in this chapter:

• How does a microservice choose the order of events to process when consuming from multiple partitions?

• How does a microservice handle out-of-order and late-arriving events?

• How do we ensure that our microservices produce deterministic results when processing streams in near–real time versus when processing from the beginning of the streams?

We can answer these questions by examining timestamps, event scheduling, watermarks, and stream times, and how they contribute to deterministic processing. Bugs, errors, and changes in business logic will also necessitate reprocessing, making deterministic results important. This chapter also explores how out-of-order and late-arriving events can occur, strategies for handling them, and mitigating their impact on our workflows.

Note

This chapter is fairly information-dense despite my best ...