Frank Kane's Taming Big Data with Apache Spark and Python

The SparkContext object

From a programming standpoint, you're going to be given a SparkContext object. Now remember when we first tested out Spark just to see if it was running, we ran something called Pyspark, which gave us an interactive shell for Spark, and that gave us an sc object automatically, which we could use to run operations on. Once you have that SparkContext object, you can use that for creating RDDs; it's a little bit more useful, obviously, to have a script you can actually run over and over again. So, in a case where you're actually running a stand-alone script, you will create that sc object by hand and initialize it the way you want to. Either way, sc stands for SparkContext, and from a programming standpoint, it is sort ...