Chapter 2. Using the Spark Shell

The Spark shell is a wonderful tool for rapid prototyping with Spark. It helps to be familiar with Scala, but that isn't necessary. The Spark shell works with Scala and Python. The Spark shell allows you to interactively query and communicate with the Spark cluster. This can be great for debugging, for just trying things out, or interactively exploring new datasets or approaches. The previous chapter should have gotten you to the point of having a Spark instance running, so now, all you need to do is start your Spark shell and point it at your running instance with the command given in the next few lines. Spark will start an instance when you invoke the Spark shell or start a Spark program from an IDE. So, a local ...

Get Fast Data Processing with Spark - Second Edition now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.