Frank Kane's Taming Big Data with Apache Spark and Python

Specifying memory per executor

Like we talked about earlier, we're going to use the default empty SparkConf in the driver script. That way we'll use the defaults that Elastic MapReduce sets up for us, and that will automatically tell Spark that it should be running on top of EMR's Hadoop cluster manager. Then it will automatically know what the layout of the cluster is, who the master is, how many client machines I have, who they are, how many executors they have, and so on. Now, when we're actually running this, we're going to pass one extra argument to spark-submit. In the past, we always just called spark-submit followed by the script name and whatever parameters we were passing into it. However, there's a bunch of other options you can ...