Running Spark in Mesos

Mesos is an open source cluster manager started as a UC Berkley research project in 2008 and quite widely used by a number of organizations. Spark supports Mesos, and Matei Zahria has given a key note at Mesos Con in June of 2016. Here is a link to the YouTube video of the keynote: http://bit.ly/2huK1aC.

Before you start

If you haven't installed Mesos previously, the getting started page on the Apache website gives a good walk through of installing Mesos on Windows, MacOS, and Linux. Follow the URL https://mesos.apache.org/gettingstarted/.

  1. Once installed you need to start-up Mesos on your cluster
  2. Starting Mesos Master: ./bin/mesos-master.sh -ip=[MasterIP] -work-dir=/var/lib/mesos
  3. Start Mesos Agents on all your worker nodes: ...

Get Learning Apache Spark 2 now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.