Big Data

Chapter 16. Micro-batch stream processing

This chapter covers

• Exactly-once processing semantics
• Micro-batch processing and its trade-offs
• Extending pipe diagrams for micro-batch stream processing

You’ve learned the main concepts of the speed layer in the last four chapters: realtime views, incremental algorithms, stream processing, and how all those fit together. There are no more fundamental concepts to learn about the speed layer—instead, in this chapter we’ll focus on a different method of stream processing that makes certain trade-offs to get benefits like improved accuracy and higher throughput.

The one-at-a-time stream processing you’ve learned is very low latency and simple to understand. But it can only provide an at-least-once ...