Mastering Hadoop 3

Spark workers

The Spark workers are responsible for managing executors running on their own machine and making communication with the master node. They are listed as follows:

• Backend process: The backend processes running on each worker node is responsible for launching the executor.
• Executors: Each executor contains a thread pool where each thread executes tasks in parallel. Executors read and process the data from a file and writes data to the target location. 
• Cache: Executors also contains the cache area, which can be used to persist RDD in-memory. Caching helps in optimizing jobs that require an iterative approach. It may not be necessary that the entire RDD fits in memory, and so we must choose the right caching strategy to make sure ...