Errata

First, sorry that I couldn't give the page number as I am reading the book over Safari and I couldn't see a page number. This is just a minor issue that's located under the "3. The Hadoop Distributed Filesystem" section and in the "Data Flow" subsection when describing the "Anatomy of a File Write". And the issue is in the paragraph starting with "As the client writes data (step 3)".

In this paragraph it says that "The list of datanodes forms a pipeline, and here we’ll assume the replication level is three, so there are three nodes in the pipeline. ". I think it would be technically more correct if this sentence says "min replication level" instead of "replication level", because during a file write only "min replication number" of nodes form a pipeline and get written synchronously (according to the figure number 4 shows a synchronous write pipeline), the remaining replicas (that is replication level - min replication level) are updated asynchronously after the write succeeds. In fact this is already mentioned in the following paragraphs in this section. So this is just a minor issue to make the sentence less confusing as when it says "replication level" the reader can easily take it as the value of the "dfs.replication" parameter while this sentence really means the value of the "dfs.namenode.replication.min" parameter.

Nezih

The command for streaming using Ruby files names the full path of the mapper, combiner, and reducer. The command seems to work only when the base names are used.

% hadoop jar /usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-streaming.jar -files ch02-mr-intro/src/main/ruby/max_temperature_map.rb,ch02-mr-intro/src/main/ruby/max_temperature_reduce.rb -input input/ncdc/all -output output -mapper max_temperature_map.rb -combiner max_temperature_reduce.rb -reducer max_temperature_reduce.rb

The commands for running the ConfigurationPrinter were listed in the book as:

% mvn compile
% export HADOOP_CLASSPATH=target/classes/
% hadoop ConfigurationPrinter -conf conf/hadoop-localhost.xml \
| grep yarn.resourcemanager.address=
yarn.resourcemanager.address=localhost:8032

But there is no target/ in the root project directory after running mvn compile. The second command should be:

% export HADOOP_CLASSPATH=ch06-mr-dev/target/classes/

In this query:

select station, year, avg(max_temperature)
from (
select station, year, max(temperature) as max_temperature
...
group by station, year
) mt
group by station, year;

The subquery produces a single (station, year, max_temperature) record for each (station, year) grouping ... so the outer select computes the "average" of a single temperature.

Or am I missing something?

The program in Java compares strings using the != operator, which will not work unless the strings in all the records in the RDD are interned. It should be ! rec[1].equals("9999") instead.