The Spark engine
To program with Spark, a basic understanding of Spark components is needed. In this section, some of the important Spark components along with their execution mechanism will be explained so that developers and data scientists can write programs and build applications.
Before getting into the details, we suggest you take a look at the following diagram so that the descriptions of the Spark gears are more comprehensible as you read further:
The Spark shell is an example of a driver program. A driver program is a process that executes in the JVM and runs the user's main function on it. It has a SparkContext object which ...