Kafka Connect

Chapter 7. Deploying and Operating Kafka Connect Clusters

In this chapter, we focus on how to deploy and operate Kafka Connect clusters. Starting with deployment, we look at how to build a Kafka Connect environment, customize it using connector and worker plug-ins, and determine the network and permissions requirements. We also discuss how to size a cluster efficiently to handle the workload for your use cases. Then, moving into operation, we look at the most common operations that administrators perform on a running Kafka Connect cluster, such as adding and removing workers, applying upgrades, restarting failed tasks and resetting offsets. Finally, we give an overview of the Kafka Connect REST API and explain how to use each of the available endpoints to manage and monitor clusters.

After reading this chapter, you will be able to deploy and maintain a production Kafka Connect cluster.

Preparing the Kafka Connect Environment

In Chapter 3, we talked about the difference between standalone and distributed mode when you deploy Kafka Connect. Distributed mode comes with more operating steps but is recommended for production deployments due to the added resiliency it provides. For this chapter, we assume you are deploying Kafka Connect in distributed mode.

Note

If you do choose to use standalone mode, keep the following in mind as you read this chapter:

• In standalone mode, you only deploy a single, independent worker and cannot automatically scale with new workers.

• In standalone ...