book

Kafka Connect

by Mickael Maison, Kate Stanley

September 2023

Beginner to intermediate

400 pages

10h 33m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Who Should Read This BookKafka VersionsNavigating This BookConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgements
Kafka Connect FeaturesPluggable ArchitectureScalability and ReliabilityDeclarative Pipeline DefinitionPart of Apache KafkaUse CasesCapturing Database ChangesMirroring Kafka ClustersBuilding Data LakesAggregating LogsModernizing Legacy SystemsAlternatives to Kafka ConnectSummary
A Distributed Event Streaming PlatformOpen SourceDistributedEvent StreamingPlatformKafka ConceptsPublish-SubscribeBrokers and RecordsTopics and PartitionsReplicationRetention and CompactionKRaft and ZooKeeperInteracting with KafkaProducersConsumersKafka StreamsGetting Started with KafkaStarting KafkaSending and Receiving RecordsRunning a Kafka Streams ApplicationSummary
Kafka Connect RuntimeRunning Kafka ConnectKafka Connect REST APIInstalling Plug-InsDeployment ModesSource and Sink ConnectorsConnectors and TasksConfiguring ConnectorsRunning ConnectorsConvertersData Format and SchemasConfiguring ConvertersUsing ConvertersTransformations and PredicatesTransformation Use CasesPredicatesConfiguring Transformations and PredicatesUsing Transformations and PredicatesSummary
Choosing a ConnectorPipeline DirectionLicensing and SupportConnector FeaturesDefining Data ModelsData TransformationMapping Data Between SystemsFormatting DataData FormatsSchemasExploring Kafka Connect InternalsInternal TopicsGroup MembershipRebalance ProtocolsHandling Failures in Kafka ConnectWorker FailureConnector/Task FailureKafka/External Systems FailureDead Letter QueuesUnderstanding Processing SemanticsSink ConnectorsSource ConnectorsSummary
Confluent S3 Sink ConnectorConfiguring the ConnectorExactly-Once SemanticsRunning the ConnectorConfluent JDBC Source ConnectorConfiguring the ConnectorRunning the ConnectorDebezium MySQL Source ConnectorConfiguring the ConnectorEvent FormatsRunning the ConnectorSummary
Introduction to MirroringExploring Mirroring Use CasesMirroring in PracticeIntroduction to MirrorMakerCommon ConceptsDeployment ModesMirrorMaker ConnectorsMirrorSourceConnectorMirrorCheckpointConnectorMirrorHeartbeatConnectorRunning MirrorMakerDisaster Recovery ExampleGeo-Replication ExampleSummary

Preparing the Kafka Connect EnvironmentBuilding a Kafka Connect EnvironmentInstalling Plug-InsNetworking and PermissionsWorker Plug-InsConfiguration ProvidersREST ExtensionsConnector Client Configuration Override PoliciesSizing and Planning CapacityUnderstanding Kafka Connect Resource UtilizationHow Many Workers and Tasks?Operating Kafka Connect ClustersAdding WorkersRemoving WorkersUpgrading and Applying Maintenance to WorkersRestarting Failed Tasks and ConnectorsResetting Offsets of ConnectorsAdministering Kafka Connect Using the REST APICreating and Deleting a ConnectorConnector and Task ConfigurationControlling the Lifecycle of ConnectorsListing Connector OffsetsDebugging IssuesSummary
Configuring the RuntimeConfigurations for ProductionFine-Tuning ConfigurationsConfiguring ConnectorsTopic ConfigurationsClient OverridesConfigurations for Exactly-OnceConfigurations for Error HandlingConfiguring Kafka Connect Clusters for SecuritySecuring the Connection to KafkaConfiguring PermissionsSecuring the REST APISummary
Monitoring LogsLogging ConfigurationUnderstanding Startup LogsAnalyzing LogsMonitoring MetricsMetrics ReportersAnalyzing MetricsExploring MetricsKey MetricsKafka Connect Runtime MetricsOther System MetricsSummary
Introduction to KubernetesVirtualization TechnologiesKubernetes FundamentalsRunning Kafka Connect on KubernetesContainer ImageDeploying WorkersNetworking and MonitoringConfigurationUsing a Kubernetes Operator to Deploy Kafka ConnectIntroduction to Kubernetes OperatorsKubernetes Operators for Kafka ConnectStrimziGetting a Kubernetes EnvironmentStarting the OperatorKafka Connect CRDsDeploying a Kafka Connect Cluster and ConnectorsMirrorMaker CRDSummary
Common Concepts and APIsBuilding a Custom ConnectorThe Connector APIConfigurationsThe Task APIKafka Connect RecordsThe ConnectorContext APIImplementing Source ConnectorsThe SourceTask APISource RecordsThe SourceConnectorContext and SourceTaskContext APIsExactly-Once SupportImplementing Sink ConnectorsThe SinkTask APISink RecordsThe SinkConnectorContext and SinkTaskContext APIsSummary
Implementing Connector Plug-InsThe Transformation APIThe Predicate APIThe Converter and HeaderConverter APIsImplementing Worker Plug-InsThe ConfigProvider APIThe ConnectorClientConfigOverridePolicy APIThe ConnectRestExtension APIsSummary

Content preview from Kafka Connect

Chapter 11. Building Source and Sink Connectors

As mentioned in previous chapters, the Apache Kafka project does not include any connectors apart from MirrorMaker and the example FileStream connectors. Instead, the Kafka community and many vendors build connectors. There are now hundreds of them available for the most popular databases, data processing tools, storage tools, and protocols.

However, if there are no connectors for the systems you use, or if the existing connectors are not suitable for your use cases, you can build your own source and sink connectors. In this chapter, we describe the process of implementing a connector from scratch. We cover the classes in the Kafka Connect public API that you need to implement and explain the main steps, best practices, and decisions you need to make in order to build a connector that satisfies your requirements.

Source and sink connectors share a lot of the same building blocks, so many pieces of work and decision points are the same when building your own. Before diving into the specifics of each type of connector, we look at the concepts that apply to both types, such as how to build the connector and the common APIs.

Common Concepts and APIs

As explained in previous chapters, a connector consists of two parts: the connector and the tasks. The initial entry point is the connector, which then configures one or more tasks that actually do the work. These parts map to the Connector and Task classes that you need to implement in ...