Chapter 12. Extending Kafka Connect with Connector and Worker Plug-Ins
Kafka Connect has two different kinds of plug-ins that you can use to extend its functionality:
- Connector plug-ins
Used to influence the behavior of a particular pipeline
- Worker plug-ins
Used to customize the Kafka Connect runtime
Kafka Connect comes with a few built-in plug-ins, and the Kafka community has also built many more. In this chapter, we explain how to implement your own connector and worker plug-ins. In “Implementing Connector Plug-Ins”, we omit connectors because we explain how to implement those in Chapter 11. Before considering writing your own plug-in, check to see if there is already one available for your use case.
There are a few common considerations that apply no matter which type of plug-in you are implementing. First, the building and packaging process for connector and worker plug-ins is the same as for connectors. We describe the steps in “Building a Custom Connector”. Secondly, there are some
Exception classes included in the
clients packages that you can use in your plug-in when you need to throw an
Exception. Two such classes are
ConfigException. You can either use these directly or subclass them. Finally, as for any piece of software, it’s important to test the plug-ins you implement to ensure that they work as expected.
Connector and worker plug-ins tend to be easier to test than connectors because they have much simpler lifecycles. After being ...