May 2024
Intermediate to advanced
110 pages
1h 19m
Chinese
Content preview from 面向MapReduce的Hadoop优化Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
第3章 检测系统瓶颈
如何判断Hadoop MapReduce 作业是否正在以最优的方式运行？在我们的咨询实践中，我们收到的与性能相关的最常见的请求就是找出一个特定作业运行时间过长的原因，以及找出发生上述状况的瓶颈所在。
在第1章和第2章中，我们学习了影响Hadoop MapReduce性能的因素，以及Hadoop MapReduce通用参数的设置。在本章中，我们将继续学习如何检测潜在的系统瓶颈。
本章将展现性能调优的过程，指出作业调优前创建基线的重要性，以及如何使用基线对集群进行调优。读者将学到如何识别资源瓶颈，以及为了打破这些瓶颈应该采取哪些措施。
本章将介绍以下内容：
介绍性能调优过程；
创建性能基线；
Hadoop 集群调优的方法；
识别系统资源瓶颈。
3.1 性能调优
性能调优的基本目标是确保特定集群配置中的可用资源（CPU、RAM、I/O 和网络）对特定作业可用，并以均衡的方式使用。
Hadoop MapReduce 资源分为几种类别，如计算资源、内存资源、网络带宽资源和 I/O 存储资源。上述资源中任何一种表现不好都会影响 Hadoop 的性能，造成作业运行缓慢。因此，Hadoop MapReduce性能调优的核心是让Hadoop 集群资源均衡，而不仅仅是调整几个变量。
简单来说，对Hadoop MapReduce 作业进行调优的过程是采用多种分析手段来研究 Hadoop 性能指标和指示器，从而掌握执行时间、内存使用量及本地文件系统读出和存入的字节数等。
Hadoop性能调优是一个迭代过程。首先启动作业，然后分析Hadoop计数器，调整计数器，然后重新运行作业。不断重复这一过程直到 Hadoop 集群达到最佳性能。下列步骤描述了这一过程。
1.创建一项基线，用来对整个系统进行首次性能评估。使用 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.