book

Kafka Connect

by Mickael Maison, Kate Stanley

September 2023

Beginner to intermediate

400 pages

10h 33m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Who Should Read This BookKafka VersionsNavigating This BookConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgements
Kafka Connect FeaturesPluggable ArchitectureScalability and ReliabilityDeclarative Pipeline DefinitionPart of Apache KafkaUse CasesCapturing Database ChangesMirroring Kafka ClustersBuilding Data LakesAggregating LogsModernizing Legacy SystemsAlternatives to Kafka ConnectSummary
A Distributed Event Streaming PlatformOpen SourceDistributedEvent StreamingPlatformKafka ConceptsPublish-SubscribeBrokers and RecordsTopics and PartitionsReplicationRetention and CompactionKRaft and ZooKeeperInteracting with KafkaProducersConsumersKafka StreamsGetting Started with KafkaStarting KafkaSending and Receiving RecordsRunning a Kafka Streams ApplicationSummary
Kafka Connect RuntimeRunning Kafka ConnectKafka Connect REST APIInstalling Plug-InsDeployment ModesSource and Sink ConnectorsConnectors and TasksConfiguring ConnectorsRunning ConnectorsConvertersData Format and SchemasConfiguring ConvertersUsing ConvertersTransformations and PredicatesTransformation Use CasesPredicatesConfiguring Transformations and PredicatesUsing Transformations and PredicatesSummary
Choosing a ConnectorPipeline DirectionLicensing and SupportConnector FeaturesDefining Data ModelsData TransformationMapping Data Between SystemsFormatting DataData FormatsSchemasExploring Kafka Connect InternalsInternal TopicsGroup MembershipRebalance ProtocolsHandling Failures in Kafka ConnectWorker FailureConnector/Task FailureKafka/External Systems FailureDead Letter QueuesUnderstanding Processing SemanticsSink ConnectorsSource ConnectorsSummary
Confluent S3 Sink ConnectorConfiguring the ConnectorExactly-Once SemanticsRunning the ConnectorConfluent JDBC Source ConnectorConfiguring the ConnectorRunning the ConnectorDebezium MySQL Source ConnectorConfiguring the ConnectorEvent FormatsRunning the ConnectorSummary
Introduction to MirroringExploring Mirroring Use CasesMirroring in PracticeIntroduction to MirrorMakerCommon ConceptsDeployment ModesMirrorMaker ConnectorsMirrorSourceConnectorMirrorCheckpointConnectorMirrorHeartbeatConnectorRunning MirrorMakerDisaster Recovery ExampleGeo-Replication ExampleSummary

Preparing the Kafka Connect EnvironmentBuilding a Kafka Connect EnvironmentInstalling Plug-InsNetworking and PermissionsWorker Plug-InsConfiguration ProvidersREST ExtensionsConnector Client Configuration Override PoliciesSizing and Planning CapacityUnderstanding Kafka Connect Resource UtilizationHow Many Workers and Tasks?Operating Kafka Connect ClustersAdding WorkersRemoving WorkersUpgrading and Applying Maintenance to WorkersRestarting Failed Tasks and ConnectorsResetting Offsets of ConnectorsAdministering Kafka Connect Using the REST APICreating and Deleting a ConnectorConnector and Task ConfigurationControlling the Lifecycle of ConnectorsListing Connector OffsetsDebugging IssuesSummary
Configuring the RuntimeConfigurations for ProductionFine-Tuning ConfigurationsConfiguring ConnectorsTopic ConfigurationsClient OverridesConfigurations for Exactly-OnceConfigurations for Error HandlingConfiguring Kafka Connect Clusters for SecuritySecuring the Connection to KafkaConfiguring PermissionsSecuring the REST APISummary
Monitoring LogsLogging ConfigurationUnderstanding Startup LogsAnalyzing LogsMonitoring MetricsMetrics ReportersAnalyzing MetricsExploring MetricsKey MetricsKafka Connect Runtime MetricsOther System MetricsSummary
Introduction to KubernetesVirtualization TechnologiesKubernetes FundamentalsRunning Kafka Connect on KubernetesContainer ImageDeploying WorkersNetworking and MonitoringConfigurationUsing a Kubernetes Operator to Deploy Kafka ConnectIntroduction to Kubernetes OperatorsKubernetes Operators for Kafka ConnectStrimziGetting a Kubernetes EnvironmentStarting the OperatorKafka Connect CRDsDeploying a Kafka Connect Cluster and ConnectorsMirrorMaker CRDSummary
Common Concepts and APIsBuilding a Custom ConnectorThe Connector APIConfigurationsThe Task APIKafka Connect RecordsThe ConnectorContext APIImplementing Source ConnectorsThe SourceTask APISource RecordsThe SourceConnectorContext and SourceTaskContext APIsExactly-Once SupportImplementing Sink ConnectorsThe SinkTask APISink RecordsThe SinkConnectorContext and SinkTaskContext APIsSummary
Implementing Connector Plug-InsThe Transformation APIThe Predicate APIThe Converter and HeaderConverter APIsImplementing Worker Plug-InsThe ConfigProvider APIThe ConnectorClientConfigOverridePolicy APIThe ConnectRestExtension APIsSummary

Content preview from Kafka Connect

Part II. Developing Data Pipelines with Kafka Connect

The second part of this book explains how to build data pipelines with Kafka Connect and is particularly relevant for data engineers. It describes in detail the roles of all the types of connector plug-ins and how to combine them.

It also covers the process of defining a pipeline, the main decision points to consider, and the best practices to follow. Finally, it explores the Capturing Database Changes, Building Data Lakes, and Mirroring Kafka Clusters use cases we introduced in Chapter 1 by demonstrating concrete examples with specific connectors.