Hadoop Operations

Chapter 7. Resource Management

What Is Resource Management?

In a perfect world, multiple groups of users, each with jobs with different service level agreements (SLAs), would be able to coexist on a shared system, blissfully unaware of one another. Disk space and IO, memory, CPU time, and network bandwidth would be limitless, and MapReduce jobs would run unimpeded by the mundane details of competing disk access patterns, activity oscillation throughout the work day, and poorly implemented algorithms that spew data to disk faster than your average administrator can rap a developer across the knuckles. We don’t live in that world and the infinite elastic utopia where the locality of terabytes (or more) of data is meaningless is not yet upon us. The question then, is how do we create an environment where competing interests can coexist peacefully, all the while ensuring the right jobs receive the right resources at the right time?

Resource management is about controlling how much of a finite resource should be allocated to a given user, group, or job. The resources—generally bisected into the categories of storage and compute—should be appropriately distributed across whatever designations make sense within the larger organization. This can mean allocation by the person for whom the work is performed, although more commonly, it is by line of business, where each group has jobs of varying criticality. Resource allocation is related to authentication in that we must know who someone is ...