September 2023
Beginner to intermediate
400 pages
10h 33m
English
Content preview from Kafka Connect
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Chapter 3. Components in a Kafka Connect Data Pipeline
A Kafka Connect pipeline involves one or more plug-ins and a Kafka Connect runtime that is responsible for executing them. Kafka Connect streams data between a Kafka cluster and one or more external systems. It is usual for a Kafka Connect pipeline to interact with a single Kafka cluster. For a single Kafka cluster, there is no limit to the number of Kafka Connect pipelines that it can be part of.
In this chapter, we take a closer look at the runtime and each of the Kafka Connect connector plug-ins: connectors, converters, transformations, and predicates. For each component, we explain its role in pipelines and how to use it. People often use the term “Connect” to refer to one component or the whole pipeline, so we introduce the correct terms for each component so you can differentiate them. By the end of this chapter, you will know how to build, configure and run a basic Kafka Connect pipeline using the official Kafka distribution.
Kafka Connect Runtime
At its core, Kafka Connect is a runtime that runs and manages data pipelines. You can easily run Kafka Connect on a laptop using the scripts, JAR files, and configuration files provided in the Kafka distribution. For example, Kafka 3.5.0 includes the following script in the bin directory for Unix-like operating systems:
connect-distributed.sh
The equivalent script for Windows operating systems is under bin/windows in the Kafka distribution:
connect-distributed.bat
The libs ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.