May 2025
Intermediate to advanced
402 pages
5h 9m
Chinese
Content preview from Kafka Connect
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
第 7 章. 部署和运行Kafka Connect 集群
本作品已使用人工智能进行翻译。欢迎您提供反馈和意见：translation-feedback@oreilly.com
在本章中，我们将重点介绍如何部署和操作 Kafka Connect 群集。从部署开始，我们将探讨如何构建 Kafka Connect 环境，使用连接器和 Worker 插件对其进行自定义，并确定网络和权限要求。我们还将讨论如何有效地调整集群规模，以处理使用案例的工作量。然后，进入操作阶段，我们将介绍管理员在运行中的 Kafka Connect 集群上执行的最常见操作，例如添加和删除 Worker、应用升级、重启失败的任务和重置偏移量。最后，我们将概述 Kafka Connect REST API，并解释如何使用每个可用端点来管理和监控集群。
读完本章后，你将能够部署和维护一个生产型 Kafka Connect 集群。
准备 Kafka Connect 环境
在第 3 章中，我们谈到了部署 Kafka Connect 时独立模式和分布式模式的区别。分布式模式有更多的操作步骤，但由于它提供了额外的弹性，因此推荐用于生产部署。在本章中，我们假设你在分布式模式下部署 Kafka Connect。
备注
如果您选择使用独立模式，请在阅读本章时牢记以下几点：
在独立模式下，，只能部署单个独立的工作员，并且不能随着新工作员的加入而自动扩展。
在单机模式下，工作者将状态存储在文件系统中。在考虑大小和容量规划时，请记住这一点。
在决定如何配置 Kafka Connect 集群之前，必须先选择部署它的环境。与 Kafka 一样，Kafka Connect 也是一个基于 Java 的项目。这，意味着只要你有Kafka Connect库，就可以在任何Java环境下运行它。请确保使用Kafka 网站上推荐的 Java 版本。
Kafka Connect Worker 和 Kafka Brokers 可以运行在同一个物理节点上，也可以分开运行。将它们分开可以提供更强的故障恢复能力，但在较小的环境中，将它们放在一起很常见。无论如何放置，Kafka Connect Worker 的部署、配置和升级都应独立于相关 Kafka 集群中的 Broker。虽然 Worker 和 Broker 需要类似的环境，但它们的生命周期不同，需要独立扩展。
部署 Kafka Connect 的最常见方式是将其部署在与其交互的 Kafka 集群相同的基础架构上。这样做可以在两个方面带来好处：操作流程和网络流量。Kafka 和 Kafka Connect 的操作流程类似。它们都需要 Java 环境，都是分布式工作负载，可以类似的方式进行扩展和监控。这意味着，无论你是将它们部署在裸机上，还是部署在 Kubernetes 这样的平台上，都更容易对两者使用相同的基础架构，而不必为多个环境采用或创建工具。Kafka Connect 使用 Kafka 来存储状态并在 Worker 之间进行协调。如果集群之间距离较近，这种流量的延迟会更低。
在某些情况下，最好与外部系统一起运行 Kafka Connect，而不是与 Kafka 集群一起运行。这取决于外部系统和连接器。例如，消息队列系统 IBM MQ 包括两种连接选项：绑定模式和客户端模式。IBM MQ 连接器支持这两种模式，但对于只允许绑定连接的 MQ 安装，Kafka Connect 集群和连接器必须与 IBM MQ 系统在同一环境中运行，连接才能成功。
让我们来看看为 Kafka ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.