Chapter 6. How MapReduce Works
In this chapter, we look at how MapReduce in Hadoop works in detail. This knowledge provides a good foundation for writing more advanced MapReduce programs, which we will cover in the following two chapters.
Anatomy of a MapReduce Job Run
You can run a MapReduce job with a single line of code:
It’s very short, but it conceals a great deal of processing behind the
scenes. This section uncovers the steps Hadoop takes to run a
The whole process is illustrated in Figure 6-1. At the highest level, there are four independent entities:
The client, which submits the MapReduce job.
The jobtracker, which coordinates the job run. The jobtracker is a Java application whose main class is
The tasktrackers, which run the tasks that the job has been split into. Tasktrackers are Java applications whose main class is
The distributed filesystem (normally HDFS, covered in Chapter 3), which is used for sharing job files between the other entities.
Figure 6-1. How Hadoop runs a MapReduce job
JobClient is a convenience
method that creates a new
JobClient instance and
submitJob() on it (step 1 in
Figure 6-1). Having submitted
runJob() polls the job’s progress once a second and reports the progress to the console if it has changed since the last report. When the job is complete, ...