January 2018
Beginner to intermediate
234 pages
7h 8m
Chinese
Content preview from Kafka权威指南
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
构建数据管道
｜
105
因为
Kafka
会长时间地保留数据，所以我们可以在适当的时候回过头来重新处理出错
的数据。
7.1.8
耦合性和灵活性
数据管道最重要的作用之一是解耦数据源和数据池。它们在很多情况下可能发生耦合。
临时数据管道
有些公司为每一对应用程序建立单独的数据管道。例如，他们使用
Logstash
向
ElasticSearch
导入日志，使用
Flume
向
HDFS
导入日志，使用
GoldenGate
将
Oracle
的
数据导到
HDFS
，使用
Informatica
将
MySQL
的数据或
XML
导到
Oracle
，等等。他们
将数据管道与特定的端点耦合起来，并创建了大量的集成点，需要额外的部署、维护和
监控。当有新的系统加入时，他们需要构建额外的数据管道，从而增加了采用新技术的
成本，同时遏制了创新。
元数据丢失
如果数据管道没有保留
schema
元数据，而且不允许
schema
发生变更，那么最终会导
致生产者和消费者之间发生紧密的耦合。没有了
schema
，生产者和消费者需要额外的
信息来解析数据。假设数据从
Oracle
流向
HDFS
，如果
DBA
在
Oracle
里添加了一个字
段，而且没有保留
schema
信息，也不允许修改
schema
，那么从
HDFS
读取数据时可能
会发生错误，因此需要双方的开发人员同时升级应用程序才能解决这个问题。不管是哪
一种情况，它们的解决方案都不具备灵活性。如果数据管道允许
schema
发生变更，应
用程序各方就可以修改自己的代码，无需担心对整个系统造成破坏。
末端处理
我们在讨论数据转换时就已提到，数据管道难免要做一些数据处理。在不同的系统之间 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.