Chapter 3. Building and Running a Spark Application

Using Spark in an interactive mode with the Spark shell has limited permanence and does not work in Java. Building Spark jobs is a bit trickier than building a normal application as all dependencies have to be available on all the machines that are in your cluster. This chapter will cover the process of building a Java and Scala Spark job with Maven or sbt (simple-build-tool) and how to build Spark jobs with a non-Maven aware build system. A reference website to build Spark is http://spark.apache.org/docs/latest/building-spark.html.

Building your Spark project with sbt

The sbt is a popular build tool for Scala that supports building both Scala and Java codes. Building Spark projects with sbt is ...

Get Fast Data Processing with Spark - Second Edition now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.