© Zubair Nabi 2016

Zubair Nabi, Pro Spark Streaming, 10.1007/978-1-4842-1479-4_6

6. The Art of Side Effects

Zubair Nabi

(1)Lahore, Pakistan

He who performs not practical work nor makes experiments will never attain to the least degree of mastery.

—Jabir ibn Hayyan (Geber)

Spark Streaming applications by design are stateless and side-effect free: running the same application an infinite number of times results in the same behavior and output. Similar to functional programming, this simplifies debugging and reasoning about the state of a program, because input and output paths are deterministic. Although side-effect-free applications have many advantages, in distributed systems side effects cannot be completely avoided, especially when interfacing ...

Get Pro Spark Streaming: The Zen of Real-Time Analytics Using Apache Spark now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.