Mastering Hadoop 3

Operating system tuning

The operating system is responsible for most of the system level tasks, such as the following:

• Transparent Huge Pages (THP): The machines that are used in Hadoop must have THP disabled. Let's understand what a THP is. In most Linux systems, the default block size is 4 KB and thus large files will have more physical blocks. The processing of files will require more blocks to be loaded to memory and thus require more iteration, which can cause slower performance. The THP allocates single memory addresses for all the blocks that are referred to as Huge Pages and thus require less iteration to read and process file. Hadoop already has a larger block size of 128 MB and blocks are not stored in contiguous memory. The blocks ...