O'Reilly logo

Fast Data Processing with Spark 2 - Third Edition by Krishna Sankar

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Building your Spark job with something else

If neither sbt nor Maven suits your needs, you may decide to use another build system. Thankfully, Spark supports building a fat JAR file with all its dependencies, which makes it easy to include in the build system of your choice. Simply, run the sbt/sbt assembly in the Spark directory and copy the resulting assembly JAR file at core/target/spark-core-assembly-1.5.2.jar to your build dependencies, and you are good to go. It is more common to use the spark-assembly-1.5.2-hadoop2.6.0.jar file. These files exist in $SPARK_HOME$/lib (if users use a prebuilt version) or in $SPARK_HOME$/ assembly/target/scala-2.10/ (if users build the source code with Maven or sbt).

Tip

No matter what your build system is, ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required