Using the Spark shell

Spark shell provides a simple way to perform interactive analysis of data. It also enables you to learn the Spark APIs by quickly trying out various APIs. In addition, the similarity to Scala shell and support for Scala APIs also lets you also adapt quickly to Scala language constructs and make better use of Spark APIs.

Spark shell implements the concept of read-evaluate-print-loop (REPL), which allows you to interact with the shell by typing in code which is evaluated. The result is then printed on the console, without needing to be compiled, so building executable code.

Start it by running the following in the directory where you installed Spark:

./bin/spark-shell

Spark shell launches and the Spark shell automatically ...

Get Scala and Spark for Big Data Analytics now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.