O'Reilly logo

Hadoop 2.x Administration Cookbook by Gurmukh Singh

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

YARN containers and resource allocations

In YARN, there are many configuration parameters, which control the memory available to AM, containers, or the total memory that can be allocated for MapReduce or JVM heap size to be used and the number of CPU cores to be used for a job. This is covered in more detail in Chapter 8, Performance Tuning, but a rough idea is to have one core for each container and each Mapper container should have a memory of about 1 GB and the Reducer should have a memory twice the size of the Mapper. In addition to this, each node must have about 20% spare for the operating system and Hadoop daemons.

Getting ready

For this recipe, you will again need a running cluster and should have completed the previous recipes to make sure ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required