The Spark engine
To program with Spark, a basic understanding of Spark components is needed. In this section, some of the important Spark components along with their execution mechanism will be explained so that developers and data scientists can write programs and build applications.
Before getting into the details, we suggest you take a look at the following diagram so that the descriptions of the Spark gears are more comprehensible as you read further:
Driver program
The Spark shell is an example of a driver program. A driver program is a process that executes in the JVM and runs the user's main function on it. It has a SparkContext object which ...
Get Spark for Data Science now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
