High Performance Spark, 2nd Edition

Chapter 2. How Spark Works

This chapter introduces the overall design of Spark as well as its place in the big data ecosystem. Spark used to be compared to ASF MapReduce, since Spark can also be used for distributed data processing with Hadoop.¹ Now it is also often thought of as the “scaled-up” version of pandas directly, by those not familiar with MapReduce.

As we will discuss in this chapter, Spark’s design principles are quite different from those of MapReduce. Unlike Hadoop MapReduce, Spark does not need to be run in tandem with ASF Hadoop—although it often is. Increasingly Spark is run without Hadoop on cloud providers or using Kubernetes with disaggregated storage.

Spark has inherited parts of its API, design, and supported formats from other existing computational frameworks, particularly DryadLINQ.² Spark’s internals, especially how it handles failures, differ from many traditional distributed systems.

Spark’s ability to leverage lazy evaluation within memory computations makes it particularly unique. Spark’s creators believe it to be the first high-level programming language for fast, distributed data processing.³

Since the introduction of Spark, some similar systems have been created, the most notable of which is Dask, which, while aimed at smaller (but still large) workloads, also uses lazy evaluation with recompute on failure.

The successor to the research lab that created the technology that became ASF Spark has created Ray, which is another distributed system but ...