May 2024
Intermediate to advanced
110 pages
1h 19m
Chinese
Content preview from 面向MapReduce的Hadoop优化Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
第4章 识别资源薄弱环节
每个Hadoop集群包含的机器和硬件都不同。这意味着每个Hadoop安装都要针对独特的配置进行优化。要确保 Hadoop 高效执行作业，需要检查集群，识别潜在瓶颈并最终消除瓶颈。
本章提出了识别集群薄弱环节的一些场景和技术，然后介绍一些公式帮你确定NameNode和DataNode的最优配置，最后介绍如何正确配置集群以及如何确定集群中mapper和reducer的数量。
本章将介绍以下内容：
基于一些场景检查集群薄弱环节；
识别CPU 过饱和以及不正确的mapper和reducer数量；
识别海量I/O 和网络阻塞；
集群量化并给出量化的定义；
正确配置集群；
4.1 识别集群薄弱环节
基于集群的硬件和节点的数量调节Hadoop框架配置对于提高性能是行之有效的。为了确保Hadoop框架高效利用机器硬件，而且确保正确定义了mapper和reduce的数量，需要检查系统环境以找出是否存在节点、CPU或者网络的薄弱环节，然后才能够决定是让Hadoop框架按照一组新的配置工作，还是需要对其进行优化。
在接下来的几节里，我们将考察导致作业表现不佳的常见场景。每个场景用独特的技术点展现如何识别问题。这些场景覆盖了节点的健康状况、输入数据大小的尺寸、海量I/O和网络流量、不充分并发任务及CPU过饱和（CPU过饱和发生在低优先级的任务不得不等待正在运行的占据CPU的高优先级任务，而且没有其他CPU处理其他工作的时候）。
4.1.1 检查Hadoop集群节点的健康状况
在一个多节点 Hadoop 集群上，由于硬件故障，如硬盘故障或电源故障，其中一个节点可能会失效。这种情况下，这一节点不能再被 Hadoop 用来处理作业，并被框架标记为Blacklisted（黑名单）、GrayListed（灰名单）或者Excluded（被排除）。这也可以发生在单节点集群上。这种情况下，节点停机导致MapReduce作业不能得到处理。检查Hadoop是否在使用集群的所有节点是获取集群整体性能的第一步。 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.