97 Things Every Data Engineer Should Know

Chapter 63. Streaming Is Different from Batch

Dean Wampler, PhD

Many organizations have a range of sophisticated analytics using tools like Hadoop, SAS, and data warehouses. These batch processes have the disadvantage of a potentially long delay between the arrival of new data and the extraction of useful information from it, which can be a competitive disadvantage. So why not move these analytics to streaming services? Why not process new data immediately and minimize delays?

Several challenges make this transition hard. First, different tools may be required. Databases may have suboptimal read and write performance for continuous streams. Use a log-oriented system as your data backplane, like Apache Kafka or Apache Pulsar, to connect streaming services, sources, and sinks.

Familiar analysis concepts also require rethinking. For example, what does a SQL GROUP BY query mean in a streaming context, when data keeps coming and will never stop? While SQL might seem to be a poor fit for streaming, it has emerged as a popular tool for streaming logic once you add windowing, for example over event time. A GROUP BY over windows makes sense, because the window is finite. It also enables large batch jobs to be decomposed into the small, fast increments necessary for streaming.

Now you’re ready to deploy, but streaming services are harder to keep healthy compared to batch jobs. For comparison, ...