High Performance Spark, 2nd Edition

Appendix E. Spark Streaming

Cowritten with Tomasz Magdanski

Spark Streaming has two modes of running, micro-batching and continuous, with very different performance characteristics. Spark Streaming exposes both DStreams and Streaming Datasets, which are similar to RDDs and Datasets, respectively, but with some additional considerations. We will focus on Streaming Datasets in this appendix, but know if you want RDD-like semantics, they can be available with the classic DStream API. Modern Spark refers to this Dataset-based API as Structured Streaming, and this is where virtually all new development occurs.

In both modes, since streaming applications generally run for longer than batch applications, sometimes without any normal ending time, and on a long enough timeline, the survival rate for a Spark driver drops to zero. Additionally, any small resource leakage in the main component of your driver will eventually balloon up. This applies equally to executor-side leaks for continuous mode, where long-lived tasks run indefinitely.

Despite these additional considerations and constraints, you can largely use the same Spark code between streaming and “regular” applications. This is especially appealing for those who want close-to-real-time updates with the ability to backfill. In modern Structured Streaming this is commonly achieved by reusing the same query and switching triggers; for example, using Trigger.AvailableNow for batch backfill and a regular processing-time trigger for ongoing ...