In this chapter, we look at how MapReduce in Hadoop works in detail. This knowledge provides a good foundation for writing more advanced MapReduce programs, which we will cover in the following two chapters.
You can run a MapReduce job with a single line of code:
JobClient.runJob(conf). It’s very short, but it
conceals a great deal of processing behind the scenes. This section
uncovers the steps Hadoop takes to run a job.
The whole process is illustrated in Figure 6-1. At the highest level, there are four independent entities:
The client, which submits the MapReduce job.
The jobtracker, which coordinates the job run. The jobtracker
is a Java application whose main class is
The tasktrackers, which run the tasks that the job has been
split into. Tasktrackers are Java applications whose main class is
The distributed filesystem (normally HDFS, covered in Chapter 3), which is used for sharing job files between the other entities.
runJob() method on
JobClient is a convenience method that creates a
instance and calls
submitJob() on it (step 1 in
Figure 6-1). Having submitted
runJob() polls the job’s progress once a second, and reports the progress to the console if it has changed since the last report. When the job is complete, ...