Learning Hadoop 2

Protecting the filesystem metadata

Because the fsimage file is so critical to the filesystem, its loss is a catastrophic failure. In Hadoop 1, where the NameNode was a single point of failure, the best practice was to configure the NameNode to synchronously write the fsimage and edits files to both local storage plus at least one other location on a remote filesystem (often NFS). In the event of NameNode failure, a replacement NameNode could be started using this up-to-date copy of the filesystem metadata. The process would require non-trivial manual intervention, however, and would result in a period of complete cluster unavailability.

Secondary NameNode not to the rescue

The most unfortunately named component in all of Hadoop 1 was the Secondary ...