Apache Spark Quick Start Guide

Stage

A stage is nothing but a group of tasks that Spark can execute together without involving a shuffle.A stage boundary is decided based on the nature of transformations we use in our applications. In Chapter 3, Spark RDD, we discussed wide transformations and narrow transformations. Wide transformations are the ones that shuffle the data across a network, such as groupByKey, Join, and Distinct. These transformations decide the number of stages in a Spark job. In our example.py file, we are using groupBy() transformation to find out the total amount by each country and orderBy() to order the data by total amount, and therefore we should have three stages for our job. We can view the stage information using an explain() operation on an ...