Chapter 12. Running on YARN and Mesos

This chapter covers

  • YARN architecture
  • YARN resource scheduling
  • Configuring and running Spark on YARN
  • Mesos architecture
  • Mesos resource scheduling
  • Configuring and running Spark on Mesos
  • Running Spark from Docker

We examined a Spark standalone cluster in the previous chapter. Now it’s time to tackle YARN and Mesos, two other cluster managers supported by Spark. They’re both widely used (with YARN still more widespread) and offer similar functionalities, but each has its own specific strengths and weaknesses. Mesos is the only cluster manager supporting fine-grained resource scheduling mode; you can also use Mesos to run Spark tasks in Docker images. In fact, the Spark project was originally started ...

Get Spark in Action now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.