May 2025
Intermediate to advanced
402 pages
5h 9m
Chinese
Content preview from Kafka Connect
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
第 3 章. Kafka Connect数据管道中的组件
本作品已使用人工智能进行翻译。欢迎您提供反馈和意见：translation-feedback@oreilly.com
Kafka Connect 管道涉及一个或多个插件以及负责执行这些插件的 Kafka Connect 运行时。Kafka Connect 在 Kafka 集群和一个或多个外部系统之间传输数据流。Kafka Connect 管道通常与单个 Kafka 集群交互。对于单个 Kafka 集群来说，它可以加入的 Kafka Connect 管道数量没有限制。
在本章中，我们将仔细研究运行时和每个 Kafka Connect 连接器插件：连接器、转换器、转换和谓词。对于每个组件，我们都会解释其在管道中的作用以及如何使用。人们经常使用 "Connect "一词来指某个组件或整个管道，因此我们将介绍每个组件的正确术语，以便您区分它们。本章结束时，你将知道如何使用 Kafka 官方发行版构建、配置和运行一个基本的 Kafka Connect 管道。
Kafka 连接运行时
Kafka Connect 的核心是一个运行和管理数据管道的运行时。使用 Kafka 发行版中提供的脚本、JAR 文件和配置文件，你可以在笔记本电脑上轻松运行 Kafka Connect。例如，Kafka 3.5.0 在类 Unix 操作系统的 bin 目录中包含以下脚本：
connect-distributed.sh
Windows 操作系统的对应脚本在 Kafka 发行版中的bin/windows 下：
connect-distributed.bat
libs 目录包含 Kafka Connect 运行时 JAR 文件（名为connect-runtime-3.5.0.jar ），以及 Kafka Connect API JAR 文件和一些插件 JAR。
最后，config 目录包含示例属性文件，其中connect-distributed.properties, 用于启动 Kafka Connect，connect-log4j.properties, 包含 Kafka Connect 的默认日志配置。
运行 Kafka Connect
在启动 Kafka Connect 之前，，确保你已经运行了 Kafka 集群。Kafka Connect 启动脚本需要一个配置文件，因此我们将使用 Kafka 发行版config 目录中的connect-distributed.properties 文件。
配置文件必须至少提供以下值：
bootstrap.servers- 以逗号分隔的 Kafka Connect 用于 Kafka 的地址列表
group.id- Kafka Connect 集群的唯一名称，Kafka Connect 运行时使用它来识别其集群中的工作者
key.converter和value.converter- 除非另有说明，否则管道中记录的键和值将使用默认转换器插件
offset.storage.topic- Kafka Connect 用来存储源连接器偏移量的主题
config.storage.topic- Kafka Connect 用来存储连接器配置的主题
status.storage.topic- Kafka Connect 用来存储连接器状态的主题
Kafka Connect 是高度可配置的，它提供了许多额外的设置，你可以用来定制它。默认配置文件只包含其中的一小部分；我们将在第 8 章介绍完整的设置列表。
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.