Architecting HBase Applications

Chapter 17. Timeouts and Garbage Collection

Garbage collection (GC) is the process that cleans up the memory no longer referenced by Java objects. This is done to free up the memory and make it available again to the application. Garbage collections are a normal, unavoidable occurrence in Java, but they can be managed. Depending on the garbage collection algorithm used, there might be one or more garbage collection events going. The most impactful garbage collection event for an HBase server is when the Java Virtual Machine needs to perform a full garbage collection (full GC). Such an operation is known as a “stop-the-world pause” and will pause the JVM while it is cleaning enough memory to allocate required objects. During this operation, the current running threads are in a holding pattern until the JVM completes its GC. Long-running GC pauses are usually the main cause of timeouts on an HBase server, but we will also look at a few other risks.

Consequences

As stated before, full GCs are a “stop-the-world pause,” meaning any operation running on an HBase server will be paused until GC completion. These pauses can manifest themselves externally as visible big spikes in the reads and writes latencies, or even in the worst cases as false server failures. There are other issues that can reflect similar performance spikes in the RegionServer. The sources of these spikes should always be clearly identified before starting to treat GC issues. The JVM will be paused while performing ...