Hadoop: Data Processing and Modelling

Time for action – adding a rack awareness script

We can enhance the default flat rack configuration by creating a script that derives the rack location for each host.

1. Create a script in the Hadoop user's home directory on the NameNode host called rack-script.sh, containing the following text. Remember to change the IP address to one of your HDFS nodes.

   #!/bin/bash
   
   if [ $1 = "10.0.0.101" ]; then
       echo -n "/rack1 "
   else
       echo -n "/default-rack "
   fi

2. Make this script executable.

   $ chmod +x rack-script.sh
   

3. Add the following property to core-site.xml on the NameNode host:

   <property>
   <name>topology.script.file.name</name>
   <value>/home/Hadoop/rack-script.sh</value>
   </property>

4. Restart HDFS.

   $ start-dfs.sh
   

5. Check the filesystem via fsck.

   $ Hadoop fsck –rack
   

   The ...