O'Reilly logo

Fast Data Processing with Spark 2 - Third Edition by Krishna Sankar

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

iPython

Finally, let's fire up iPython and interact with the SparkContext object. As mentioned in Chapter 3, Building and Running a Spark Application, refer to the iPython site (http://jupyter.readthedocs.org/en/latest/install.html) for installing the Jupyter and iPython system.

First, change the directory to fdps-v3, where you would have downloaded the code and data for this book:

cd ~/fdps-v3

The command to start iPython is as follows:

PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook" ~/Downloads/spark-2.0.0/bin/pyspark

The iPython notebook will be launched in the web browser, as shown in the following screenshot, and you will see a list of iPython notebooks:

Click on the 000-PreFlightCheck.ipynb notebook:

Run the first cell ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required