O'Reilly logo

Hadoop 2.x Administration Cookbook by Gurmukh Singh

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Hadoop best practices

In this section, we will cover some of the common best practices for the Hadoop cluster in terms of log management and troubleshooting tools.

These are not from a tuning perspective, but to make things easier to troubleshoot and diagnose.

Things to keep in mind:

  1. Always enable logs for each daemon that runs in the Hadoop cluster. Keep the logging level to INFO and, when needed, change it to DEBUG. Once the troubleshooting is done, revert to level INFO.
  2. Implement log rotation and retention polices to manage the logs.
  3. Use tools such as Nagios to alert for any errors in the cluster before it becomes an issue.
  4. Use log aggregation and analysis tools such as Splunk to parse logs.
  5. Never co-locate the logs disk with other data disks in ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required