Kafka Connect

Chapter 12. Extending Kafka Connect with Connector and Worker Plug-Ins

Kafka Connect has two different kinds of plug-ins that you can use to extend its functionality:

Connector plug-ins

Used to influence the behavior of a particular pipeline

Worker plug-ins

Used to customize the Kafka Connect runtime

Kafka Connect comes with a few built-in plug-ins, and the Kafka community has also built many more. In this chapter, we explain how to implement your own connector and worker plug-ins. In “Implementing Connector Plug-Ins”, we omit connectors because we explain how to implement those in Chapter 11. Before considering writing your own plug-in, check to see if there is already one available for your use case.

There are a few common considerations that apply no matter which type of plug-in you are implementing. First, the building and packaging process for connector and worker plug-ins is the same as for connectors. We describe the steps in “Building a Custom Connector”. Secondly, there are some Exception classes included in the connect-api and clients packages that you can use in your plug-in when you need to throw an Exception. Two such classes are DataException and ConfigException. You can either use these directly or subclass them. Finally, as for any piece of software, it’s important to test the plug-ins you implement to ensure that they work as expected.

Connector and worker plug-ins tend to be easier to test than connectors because they have much simpler lifecycles. After being ...