Configuring MapReduce for performance
In this recipe, we will touch upon MapReduce parameters and see how we can optimize them.
For this recipe, you will again need a running cluster with HDFS and YARN. Users must have completed the recipe Configuring YARN for performance recipe.
How to do it...
- Connect to the master node
master1.cyrus.comand switch to the
- The file where these changes will be made is
- The first thing to adjust is to sort the buffer according to the HDFS block size. It must always be greater than the value of
dfs.blocksize. This can be configured as follows:
<property> <name>mapreduce.task.io.sort.mb</name> <value>200</value> </property>
- The next value to tune is the number of streams to merge ...