Cloud Native Development Patterns and Best Practices

Solution

Implement one or more consumers that in aggregate consume all events from all streams and store the unaltered, raw events in highly durable blob storage. The consumers are optimized to store the events in batches. Batching increases throughput to ensure that the consumers keep pace with event volumes, it minimizes the potential for errors that could result in the loss of data, and it optimizes the objects for later retrieval. The objects are stored with the following path: stream-name/yyyy/mm/dd/hh.

It is critical to monitor these consumers for errors, alert on their iterator age, and take timely action in the case of an interruption. These consumers are all important, as any other consumer can be repaired from the events in the ...