O'Reilly logo

Hadoop 2.x Administration Cookbook by Gurmukh Singh

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Control block report storm

When Datanodes come up in large clusters with more than 200 nodes, Namenode will be overwhelmed by the block reports and this can cause Namenode to become unresponsive.

Getting ready

This recipe makes more sense for large clusters, not in terms of the number of nodes, but the number of blocks in the cluster.

How to do it...

  1. ssh to Namenode and edit the hdfs-site.xml file to add the following property to it:
    <property>
    <name>dfs.blockreport.initialDelay</name>
    <value>20</value>
    </property>
  2. Copy hdfs-site.xml across all nodes in the cluster.
  3. Restart HDFS daemons across the nodes for the property to take effect:
    $ stop-dfs.sh
    $ start-dfs.sh
    

How it works...

The dfs.blockreport.initialDelay parameter specifies the time in seconds. ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required