HBase: The Definitive Guide

Chapter 12. Cluster Administration

Once a cluster is in operation, it may become necessary to change its size or add extra measures for failover scenarios, all while the cluster is in use. Data should be backed up and/or moved between distinct clusters. In this chapter, we will look how this can be done with minimal to no interruption.

Operational Tasks

This section introduces the various tasks necessary while operating a cluster, including adding and removing nodes.

Node Decommissioning

You can stop an individual region server by running the following script in the HBase directory on the particular server:

$ ./bin/hbase-daemon.sh stop regionserver

The region server will first close all regions and then shut itself down. On shutdown, its ephemeral node in ZooKeeper will expire. The master will notice that the region server is gone and will treat it as a crashed server: it will reassign the regions the server was carrying.

Disabling the Load Balancer Before Decommissioning a Node

If the load balancer runs while a node is shutting down, there could be contention between the load balancer and the master’s recovery of the just-decommissioned region server. Avoid any problems by disabling the balancer first: use the shell to disable the balancer like so:

hbase(main):001:0> balance_switch false
true
0 row(s) in 0.3590 seconds

This turns the balancer off. To reenable it, enter the following:

hbase(main):002:0> balance_switch true
false
0 row(s) in 0.3590 seconds

A downside to this method of stopping ...