Running the Spark Job on Mesos (local)
Unlike the Spark standalone cluster manager, which can run only Spark apps, Mesos is a cluster manager that can run a wide variety of applications, including Python, Ruby, or Java EE applications. It can also run Spark jobs. In fact, it is one of the popular go-to cluster managers for Spark. In this recipe, we'll see how to deploy our Spark application on the Mesos cluster. The prerequisite for this recipe is a running HDFS cluster.
How to do it...
Running a Spark job on Mesos is very similar to running it against the standalone cluster. It involves the following steps:
- Installing Mesos.
- Starting the Mesos master and slave.
- Uploading the Spark binary package and the dataset to HDFS.
- Running the job.