September 2023
Beginner to intermediate
400 pages
10h 33m
English
Content preview from Kafka Connect
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Chapter 5. Connectors in Action
After having covered how Kafka Connect works and how to use it, it’s time to put your knowledge into practice! In this chapter we look at, and run, some of the most popular connectors: a sink connector for Amazon Simple Storage Service (S3), a JDBC source connector, and a MySQL source connector. We explain the use cases they target and their most important configurations, and demonstrate how to use them in various scenarios.
These three connectors address common use cases and appear in many pipelines across all industries, so having a good understanding of them is valuable. Even if you don’t use these specific connectors, we expect many of the topics covered to be applicable to others.
All of the examples assume that you have a Kafka cluster running with a bootstrap server accessible at localhost:9092.
Confluent S3 Sink Connector
One of the most common use cases of Kafka Connect is to export data from Kafka into a storage system. Often, you need to keep data long after it has been processed; it could be for legal reasons, for preserving historical data, or simply for batch-oriented systems that only run periodically. While Kafka can store data indefinitely, if you handle very large amounts of data it can become costly to store it in Kafka forever.
Cloud storage systems like Amazon S3 are designed for storing large amounts of data for long durations at a low cost per gigabyte. In addition, storage services can be used as data lakes due to their ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.