book

Kafka Connect

by Mickael Maison, Kate Stanley

September 2023

Beginner to intermediate

400 pages

10h 33m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Who Should Read This BookKafka VersionsNavigating This BookConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgements
Kafka Connect FeaturesPluggable ArchitectureScalability and ReliabilityDeclarative Pipeline DefinitionPart of Apache KafkaUse CasesCapturing Database ChangesMirroring Kafka ClustersBuilding Data LakesAggregating LogsModernizing Legacy SystemsAlternatives to Kafka ConnectSummary
A Distributed Event Streaming PlatformOpen SourceDistributedEvent StreamingPlatformKafka ConceptsPublish-SubscribeBrokers and RecordsTopics and PartitionsReplicationRetention and CompactionKRaft and ZooKeeperInteracting with KafkaProducersConsumersKafka StreamsGetting Started with KafkaStarting KafkaSending and Receiving RecordsRunning a Kafka Streams ApplicationSummary
Kafka Connect RuntimeRunning Kafka ConnectKafka Connect REST APIInstalling Plug-InsDeployment ModesSource and Sink ConnectorsConnectors and TasksConfiguring ConnectorsRunning ConnectorsConvertersData Format and SchemasConfiguring ConvertersUsing ConvertersTransformations and PredicatesTransformation Use CasesPredicatesConfiguring Transformations and PredicatesUsing Transformations and PredicatesSummary
Choosing a ConnectorPipeline DirectionLicensing and SupportConnector FeaturesDefining Data ModelsData TransformationMapping Data Between SystemsFormatting DataData FormatsSchemasExploring Kafka Connect InternalsInternal TopicsGroup MembershipRebalance ProtocolsHandling Failures in Kafka ConnectWorker FailureConnector/Task FailureKafka/External Systems FailureDead Letter QueuesUnderstanding Processing SemanticsSink ConnectorsSource ConnectorsSummary
Confluent S3 Sink ConnectorConfiguring the ConnectorExactly-Once SemanticsRunning the ConnectorConfluent JDBC Source ConnectorConfiguring the ConnectorRunning the ConnectorDebezium MySQL Source ConnectorConfiguring the ConnectorEvent FormatsRunning the ConnectorSummary
Introduction to MirroringExploring Mirroring Use CasesMirroring in PracticeIntroduction to MirrorMakerCommon ConceptsDeployment ModesMirrorMaker ConnectorsMirrorSourceConnectorMirrorCheckpointConnectorMirrorHeartbeatConnectorRunning MirrorMakerDisaster Recovery ExampleGeo-Replication ExampleSummary

Preparing the Kafka Connect EnvironmentBuilding a Kafka Connect EnvironmentInstalling Plug-InsNetworking and PermissionsWorker Plug-InsConfiguration ProvidersREST ExtensionsConnector Client Configuration Override PoliciesSizing and Planning CapacityUnderstanding Kafka Connect Resource UtilizationHow Many Workers and Tasks?Operating Kafka Connect ClustersAdding WorkersRemoving WorkersUpgrading and Applying Maintenance to WorkersRestarting Failed Tasks and ConnectorsResetting Offsets of ConnectorsAdministering Kafka Connect Using the REST APICreating and Deleting a ConnectorConnector and Task ConfigurationControlling the Lifecycle of ConnectorsListing Connector OffsetsDebugging IssuesSummary
Configuring the RuntimeConfigurations for ProductionFine-Tuning ConfigurationsConfiguring ConnectorsTopic ConfigurationsClient OverridesConfigurations for Exactly-OnceConfigurations for Error HandlingConfiguring Kafka Connect Clusters for SecuritySecuring the Connection to KafkaConfiguring PermissionsSecuring the REST APISummary
Monitoring LogsLogging ConfigurationUnderstanding Startup LogsAnalyzing LogsMonitoring MetricsMetrics ReportersAnalyzing MetricsExploring MetricsKey MetricsKafka Connect Runtime MetricsOther System MetricsSummary
Introduction to KubernetesVirtualization TechnologiesKubernetes FundamentalsRunning Kafka Connect on KubernetesContainer ImageDeploying WorkersNetworking and MonitoringConfigurationUsing a Kubernetes Operator to Deploy Kafka ConnectIntroduction to Kubernetes OperatorsKubernetes Operators for Kafka ConnectStrimziGetting a Kubernetes EnvironmentStarting the OperatorKafka Connect CRDsDeploying a Kafka Connect Cluster and ConnectorsMirrorMaker CRDSummary
Common Concepts and APIsBuilding a Custom ConnectorThe Connector APIConfigurationsThe Task APIKafka Connect RecordsThe ConnectorContext APIImplementing Source ConnectorsThe SourceTask APISource RecordsThe SourceConnectorContext and SourceTaskContext APIsExactly-Once SupportImplementing Sink ConnectorsThe SinkTask APISink RecordsThe SinkConnectorContext and SinkTaskContext APIsSummary
Implementing Connector Plug-InsThe Transformation APIThe Predicate APIThe Converter and HeaderConverter APIsImplementing Worker Plug-InsThe ConfigProvider APIThe ConnectorClientConfigOverridePolicy APIThe ConnectRestExtension APIsSummary

Content preview from Kafka Connect

Part I. Introduction to Kafka Connect

The first part of this book gives a high-level introduction to Kafka Connect. It is aimed at data engineers, architects, and site reliability engineers who are unsure what Kafka Connect is or whether it is the right tool for them.

This part explains Kafka Connect’s key features and why it is so popular. It covers the most common use cases and lists some alternatives. Finally, it provides a basic introduction to Apache Kafka to help you gain the minimum level of Kafka knowledge needed to fully use Kafka Connect. This includes key terms and concepts and how to set up a simple deployment.