Metadata management

HDFS stores a large amount of structured and unstructured data in various formats. While the data is continuously growing to terabytes and petabytes, and your data is being used by Hadoop, you are likely to come across questions, such as what data is available on HDFS, how it is being used, and what type of users are using the data, the data creation timeline, and so on. Well-maintained metadata information can effectively answer these questions and thus improve the usability of the data store over HDFS.

NameNode keeps the complete fsimage in memory so that all the metadata information requests can be served in the smallest amount of time possible and persist fsimage and edit logs on the disk. fsimage contains HDFS directory ...

Get Mastering Hadoop 3 now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.