May 2025
Intermediate to advanced
402 pages
5h 9m
Chinese
Content preview from Kafka Connect
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
第 5 章 行动中的连接器
本作品已使用人工智能进行翻译。欢迎您提供反馈和意见：translation-feedback@oreilly.com
在介绍了 Kafka Connect 的工作原理和使用方法后，是时候将所学知识付诸实践了！在本章中，我们将了解并运行一些最常用的连接器：亚马逊简单存储服务（S3）的汇连接器、JDBC 源连接器和 MySQL 源连接器。我们将解释它们的目标用例和最重要的配置，并演示如何在各种场景中使用它们。
这三个连接器涉及常见用例，出现在各行各业的许多管道中，因此充分了解它们很有价值。即使您不使用这些特定的连接器，我们也希望所涉及的许多主题适用于其他连接器。
所有示例都假定你有一个 Kafka 集群正在运行，其 Bootstrap 服务器的访问地址为localhost:9092 。
Confluent S3 Sink 连接器
Kafka Connect 最常见的用例之一是从 Kafka 导出数据到存储系统。通常，您需要在数据处理后长期保存数据；这可能是出于法律原因，也可能是为了保存历史数据，或者仅仅是为了定期运行的面向批处理的系统。虽然 Kafka 可以无限期地存储数据，但如果您处理的数据量非常大，那么将数据永远存储在 Kafka 中的成本会变得很高。
Amazon S3 等云存储系统专为长时间存储大量数据而设计，每千兆字节成本低廉。此外，由于存储服务与数据处理工具集成，因此可用作数据湖。建立一个将数据从 Kafka 复制到存储服务的管道很常见。如果想把数据放回 Kafka，可以用源连接器导入。
在本节中，我们将研究 Confluent S3 汇连接器，该连接器可从Confluent Hub 网站获取。示例和配置说明使用的是该连接器的 10.5.1 版本。社区构建了多种 S3 汇连接器供你选择；每种连接器都有自己的特性和配置设置，但总体上功能相似。您可以从 Confluent Hub 下载包含 Confluent 连接器所有库的 ZIP 文件。下载完成后，解压缩其中的内容，然后将文件夹添加到 Kafka Connect 运行时的plugin.path 。
让我们先来看看这个连接器的配置和各种运行方式。
配置连接器
Confluent S3 sink 连接器可通过 进行配置，并提供 50 多种设置来微调其行为。其中大多数设置都有合理的默认值，而且大多数高级设置都是可选的，因此您通常只需设置少量配置。在本节中，我们将介绍最重要的设置，并解释它们是如何实现常见用例的。在了解基础知识后，我们建议您查看完整的设置列表，看看您的用例是否能从高级调整中受益。请注意，您可以为一些设置提供自定义实现，这些设置会显著改变连接器的工作方式。
作为汇接口，可处理数据的最大任务数等于topics 或topics.regex 中所列主题的分区数。如果将tasks.max 设为更大的值，接口仍会启动该数量的任务，但额外的任务不会处理记录，而且会不必要地占用资源。
这些配置可分为五类：
-
连接和 S3 详情
-
对象分区
-
对象命名
-
对象格式
-
对象上传
连接和 S3 详情
要将连接器连接到您的 S3 实例，首先需要您的亚马逊网络服务（AWS）凭据：访问密钥和秘钥。向连接器提供凭据有多种方法。第一种方法是同时设置aws.access.key.id 和aws.secret.access.key 。这样就可以为运行的每个 S3 连接器使用不同的凭据。另一种方法是使用凭据提供者在 Kafka Connect Worker 中设置凭据，供所有 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.