February 2021
Intermediate to advanced
432 pages
11h 7m
English
Content preview from Mastering Kafka Streams and ksqlDB
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Chapter 9. Data Integration with ksqlDB
The first step of building a stream processing application with ksqlDB is to consider where the data you want to process currently lives, and where the enriched/transformed data will eventually be written. Since ksqlDB leverages Kafka Streams under the hood, the direct input and outputs of the application you build will always be Kafka topics. ksqlDB makes it simple to integrate other data sources as well, including from popular third-party systems like Elasticsearch, PostgreSQL, MySQL, Google PubSub, Amazon Kinesis, MongoDB, and hundreds of others.
Of course, if your data already lives in Kafka and you don’t plan on writing the processed results to an external system, then working with ksqlDB’s data integration features (which are driven by Kafka Connect) isn’t required. However, should you ever need to read from/write to external systems, this chapter will provide the necessary foundations to help you connect the appropriate data sources and sinks using ksqlDB and Kafka Connect.
This chapter isn’t a comprehensive guide on Kafka Connect, which is a separate API in the Kafka ecosystem and, accordingly, a topic about which much can and has been written. We will provide enough background to get you started, and look at ksqlDB’s high-level abstractions for working with the Connect API. Some of the topics we will explore in this chapter include:
A quick Kafka Connect overview
Kafka Connect integration modes
Configuring Kafka Connect workers ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.