January 2018
Beginner to intermediate
234 pages
7h 8m
Chinese
Content preview from Kafka权威指南
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
构建数据管道
｜
103
7.1.3
高吞吐量和动态吞吐量
为了满足现代数据系统的要求，数据管道需要支持非常高的吞吐量。更重要的是，在某些
情况下，数据管道还需要能够应对突发的吞吐量增长。
由于我们将
Kafka
作为生产者和消费者之间的缓冲区，消费者的吞吐量和生产者的吞吐量
就不会耦合在一起了。我们也不再需要实现复杂的回压机制，如果生产者的吞吐量超过了
消费者的吞吐量，可以把数据积压在
Kafka
里，等待消费者追赶上来。通过增加额外的消
费者或生产者可以实现
Kafka
的伸缩，因此我们可以在数据管道的任何一边进行动态的伸
缩，以便满足持续变化的需求。
因为
Kafka
是一个高吞吐量的分布式系统，一个适当规模的集群每秒钟可以处理数百兆的
数据，所以根本无需担心数据管道无法满足伸缩性需求。另外，
Connect API
不仅支持伸
缩，而且擅长并行处理任务。稍后，我们将会介绍数据源和数据池（
Data Sink
）如何在多
个线程间拆分任务，最大限度地利用
CPU
资源，哪怕是运行在单台机器上。
Kafka
支持多种类型的压缩，在增长吞吐量时，
Kafka
用户和管理员可以通过压缩来调整网
络和存储资源的使用。
7.1.4
数据格式
数据管道需要协调各种数据格式和数据类型，这是数据管道的一个非常重要的因素。数据
类型取决于不同的数据库和数据存储系统。你可能会通过
Avro
将
XML
或关系型数据加载
到
Kafka
里，然后将它们转成
JSON
写入
ElasticSearch
，或者转成
Parquet
写入
HDFS
，或
者转成
CSV
写入
S3
。
Kafka
和
Connect ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.