Hadoop 2.x Administration Cookbook

Benchmarking Hadoop cluster

It is important to benchmark so as to have a baseline to do comparisons after making changes. In this recipe, we will look at some of the benchmarks which can help to profile the changes committed.

Before running any tests for the changed parameters, make sure to enable verbose logging and also enable GC logs for all the components by using -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:${LOG_DIR}/gc-{component}.log-$(date +'%Y%m%d%H%M').

Getting ready

Make sure that the user has a running cluster with HDFS and YARN fully functional in a multi-node cluster.

All these tests must be run first without making any changes to the cluster and then optimizing parameters, discussed in the preceding ...