Scala and Spark for Big Data Analytics

Executors

The Executors tab uses ExecutorsListener to collect information about executors for a Spark application. An executor is a distributed agent that is responsible for executing tasks. Executors are instantiated in different ways. For example, they can be instantiated when CoarseGrainedExecutorBackend receives RegisteredExecutor message for Spark Standalone and YARN. The second case is when a Spark job is submitted to Mesos. The Mesos's MesosExecutorBackend gets registered. The third case is when you run your Spark jobs locally, that is, LocalEndpoint is created. An executor typically runs for the entire lifetime of a Spark application, which is called static allocation of executors, although you can also opt in for dynamic allocation. ...