September 2023
Beginner to intermediate
400 pages
10h 33m
English
Content preview from Kafka Connect
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Chapter 6. Mirroring Clusters with MirrorMaker
The action of copying data between two Kafka clusters is called mirroring. This term is used to distinguish this process from the term “replication” that usually refers to the way data in Kafka is shared across brokers within the cluster. However, both terms are often used by the community when talking about copying data between clusters.
The idea of mirroring data between clusters is pretty much as old as Kafka itself. At Kafka’s inception, mirroring was a feature of the broker, before it was then separated into its own tool in early 2012. The tool was a standalone application named MirrorMaker, but due to its initial design, it had a number of limitations and was hard to operate. So in 2019, a new mirroring tool based on Kafka Connect was introduced via KIP-382 called MirrorMaker2. The initial MirrorMaker tool is now deprecated since Kafka 3.0 and will be removed in Kafka 4.0 (via KIP-720); the new tool is now often simply called MirrorMaker or MM2.
In this chapter we only cover the new tool, and we refer to it as MirrorMaker. We introduce use cases that rely on mirroring, explain how the MirrorMaker connectors work, and finally demonstrate how to use them through some examples.
Introduction to Mirroring
Kafka scales very well, and it’s possible to run a single cluster with an extremely large capacity. However, in many cases it’s preferable to have multiple smaller clusters. This could be to better serve different geographies, ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.