Chapter 6. YARN – Bringing Other Paradigms to Hadoop

Yet Another Resource Negotiator (YARN) is a cluster resource management layer that was introduced in Hadoop 2.0. As we saw briefly in Chapter 1, Hadoop 2.X, YARN separates out the responsibilities of the JobTracker daemon. JobTracker was responsible for:

  • Resource arbitration within a Hadoop cluster
  • MapReduce job management

The problem with the JobTracker model was that it became the single point of failure in the compute layer of a Hadoop cluster. Any failure in JobTracker meant trashing the running jobs and starting all over again. JobTracker's singular nature also became a scaling bottleneck. All job communications, scheduling, and resource management were controlled by the JobTracker master ...

Get Mastering Hadoop now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.