Kafka Connect

Preface

Kafka Connect is an awesome tool for building reliable and scalable data pipelines. It is part of the popular Apache Kafka streaming platform, and while it may not get as much attention as the brokers, clients, or Kafka Streams, Kafka Connect is a tool to be aware of. It allows you to easily get data into and out of your Kafka clusters and even mirror data between clusters. Its pluggable design makes it possible to build powerful pipelines without writing a single line of code.

We are both passionate about sharing knowledge, whether that is through presenting at conferences, writing blog posts, or just helping out fellow Kafka enthusiasts. As a result, we have spent a lot of time chatting about both Kafka and Kafka Connect to users and developers all around the world. As Kafka is a tremendously popular technology, there are a lot of great resources available such as books, blog posts, and tutorials. Many of these do cover Kafka Connect, but we see a lack of resources that go deeper into its various use cases, configurations, and operational processes. Although Kafka Connect is not hard to start using with basic knowledge, its flexibility and range of features mean that having a deeper understanding of how it works can really make a difference.

We have both given plenty of conference talks about Kafka Connect that go beyond the basics, but there’s only so much you can fit into a 40-minute session. In writing this book, we have brought together all the knowledge we have ...