O'Reilly logo

Hadoop Operations and Cluster Management Cookbook by Shumin Guo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Configuring HDFS federation

Hadoop NameNode keeps the metadata in the main memory. When the HDFS namespace becomes large, the main memory can become a bottleneck of the cluster. HDFS federation was introduced in Hadoop for MRv2. It increases the NameNode capacity and throughput by leveraging the capacity of multiple independent NameNodes, with each NameNode hosting or managing part of the HDFS namespace.

Getting ready

Currently, only Hadoop MRv2 supports NameNode federation, so we are assuming that Hadoop MRv2 has been properly configured on all the cluster machines.

Note

We are assuming that all the configurations are making changes to the $HADOOP_CONF_DIR/hdfs-site.xml file.

How to do it...

Use the following steps to configure HDFS federation:

  1. Log ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required