May 2025
Intermediate to advanced
402 pages
5h 9m
Chinese
Content preview from Kafka Connect
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
第 11 章 构建源和汇接器 构建源极和汇极连接器
本作品已使用人工智能进行翻译。欢迎您提供反馈和意见：translation-feedback@oreilly.com
如前几章所述，除了 MirrorMaker 和FileStream 连接器示例之外，Apache Kafka 项目不包括任何连接器。相反，Kafka 社区和许多供应商构建了连接器。现在有数百种连接器可用于最流行的数据库、数据处理工具、存储工具和协议。
但是，如果您使用的系统没有 连接器，或者现有的连接器不适合您的用例，您可以构建自己的源连接器和汇连接器。在本章中，我们将介绍从头开始实现连接器的过程。我们将介绍你需要实现的 Kafka Connect 公共 API 中的类，并解释为构建一个满足你的要求的连接器所需的主要步骤、最佳实践和决策。
源连接器和汇连接器共享许多相同的构建模块，因此在构建自己的连接器时，许多工作和决策点都是相同的。在深入探讨每种类型连接器的具体细节之前，我们先看看两种类型都适用的概念，例如如何构建连接器和通用 API。
通用概念和应用程序接口
如前几章所述，连接器由两部分组成：连接器和任务。最初的入口点是连接器，然后由连接器配置一个或多个实际执行工作的任务。这些部分将映射到您需要在 Kafka Connect API 中实现的Connector 和Task 类。(例如，源连接器必须实现SourceConnector 和SourceTask ，而汇连接器必须实现SinkConnector 和SinkTask ）。在运行时，您有一个SourceConnector 或SinkConnector 类的实例和一个或多个SourceTask 或SinkTask 类的实例。
除了Connector 和Task 类之外，Kafka Connect API 还包含一组的上下文类，使连接器和任务能够与运行时交互，并深入了解运行时正在做什么或影响其行为方式。
在深入了解这些应用程序接口之前，我们先来看看如何访问所需的类并设置构建环境。
创建自定义连接器
在构建连接器时，您需要访问 的所有类都是 Kafka Connect 公共 API 的一部分，而 Kafka Connect 公共 API 是用 Java 编写的。这意味着你必须用 Java 或兼容的语言（如 Scala）实现连接器。在源代码中，这些类位于org.apache.kafka.connect 包中，然后所有这些类都作为connect-api 包的一部分可用；在创建连接器项目时，您需要添加此依赖关系才能访问这些类。在你的连接器中，你不应该依赖任何其他的 Kafka Connect 或 Kafka JAR，因为这些都不是公共 API 的一部分，可能会在没有通知的情况下发生变化。在实践中，connect-api 也带来了kafka-clients 作为传递依赖，它包含了许多用于定义配置和度量的有用类，以及许多实用工具。
您可以通过几种不同的方式获取connect-api JAR 文件。首先，该文件包含在libs 目录下的 Kafka 发布包中。其次，你可以使用 Maven 或 Gradle 等构建工具，从 Maven Central Repository 等工件库中提取该文件。
实施连接器
在构建连接器时，应遵循与任何应用程序相同的最佳实践。包括在适当的地方添加日志和度量，以便用户可以高效地运行连接器，并在遇到问题时进行调试。Kafka Connect 已经提供了 SLF4J 日志门面，因此没有必要在连接器中明确包含它。此外，您不应该在 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.