book

Kafka: The Definitive Guide, 2nd Edition

by Gwen Shapira, Todd Palino, Rajini Sivaram, Krit Petty

November 2021

Beginner to intermediate

485 pages

14h 22m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Who Should Read This BookConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
Publish/Subscribe MessagingHow It StartsIndividual Queue SystemsEnter KafkaMessages and BatchesSchemasTopics and PartitionsProducers and ConsumersBrokers and ClustersMultiple ClustersWhy Kafka?Multiple ProducersMultiple ConsumersDisk-Based RetentionScalableHigh PerformancePlatform FeaturesThe Data EcosystemUse CasesKafka’s OriginLinkedIn’s ProblemThe Birth of KafkaOpen SourceCommercial EngagementThe NameGetting Started with Kafka
Environment SetupChoosing an Operating SystemInstalling JavaInstalling ZooKeeperInstalling a Kafka BrokerConfiguring the BrokerGeneral Broker ParametersTopic DefaultsSelecting HardwareDisk ThroughputDisk CapacityMemoryNetworkingCPUKafka in the CloudMicrosoft AzureAmazon Web ServicesConfiguring Kafka ClustersHow Many Brokers?Broker ConfigurationOS TuningProduction ConcernsGarbage Collector OptionsDatacenter LayoutColocating Applications on ZooKeeperSummary
Producer OverviewConstructing a Kafka ProducerSending a Message to KafkaSending a Message SynchronouslySending a Message AsynchronouslyConfiguring Producersclient.idacksMessage Delivery Timelinger.msbuffer.memorycompression.typebatch.sizemax.in.flight.requests.per.connectionmax.request.sizereceive.buffer.bytes and send.buffer.bytesenable.idempotenceSerializersCustom SerializersSerializing Using Apache AvroUsing Avro Records with KafkaPartitionsHeadersInterceptorsQuotas and ThrottlingSummary
Kafka Consumer ConceptsConsumers and Consumer GroupsConsumer Groups and Partition RebalanceStatic Group MembershipCreating a Kafka ConsumerSubscribing to TopicsThe Poll LoopThread SafetyConfiguring Consumersfetch.min.bytesfetch.max.wait.msfetch.max.bytesmax.poll.recordsmax.partition.fetch.bytessession.timeout.ms and heartbeat.interval.msmax.poll.interval.msdefault.api.timeout.msrequest.timeout.msauto.offset.resetenable.auto.commitpartition.assignment.strategyclient.idclient.rackgroup.instance.idreceive.buffer.bytes and send.buffer.bytesoffsets.retention.minutesCommits and OffsetsAutomatic CommitCommit Current OffsetAsynchronous CommitCombining Synchronous and Asynchronous CommitsCommitting a Specified OffsetRebalance ListenersConsuming Records with Specific OffsetsBut How Do We Exit?DeserializersCustom DeserializersUsing Avro Deserialization with Kafka ConsumerStandalone Consumer: Why and How to Use a Consumer Without a GroupSummary
AdminClient OverviewAsynchronous and Eventually Consistent APIOptionsFlat HierarchyAdditional NotesAdminClient Lifecycle: Creating, Configuring, and Closingclient.dns.lookuprequest.timeout.msEssential Topic ManagementConfiguration ManagementConsumer Group ManagementExploring Consumer GroupsModifying Consumer GroupsCluster MetadataAdvanced Admin OperationsAdding Partitions to a TopicDeleting Records from a TopicLeader ElectionReassigning ReplicasTestingSummary
Cluster MembershipThe ControllerKRaft: Kafka’s New Raft-Based ControllerReplicationRequest ProcessingProduce RequestsFetch RequestsOther RequestsPhysical StorageTiered StoragePartition AllocationFile ManagementFile FormatIndexesCompactionHow Compaction WorksDeleted EventsWhen Are Topics Compacted?Summary
Reliability GuaranteesReplicationBroker ConfigurationReplication FactorUnclean Leader ElectionMinimum In-Sync ReplicasKeeping Replicas In SyncPersisting to DiskUsing Producers in a Reliable SystemSend AcknowledgmentsConfiguring Producer RetriesAdditional Error HandlingUsing Consumers in a Reliable SystemImportant Consumer Configuration Properties for Reliable ProcessingExplicitly Committing Offsets in ConsumersValidating System ReliabilityValidating ConfigurationValidating ApplicationsMonitoring Reliability in ProductionSummary

Idempotent ProducerHow Does the Idempotent Producer Work?Limitations of the Idempotent ProducerHow Do I Use the Kafka Idempotent Producer?TransactionsTransactions Use CasesWhat Problems Do Transactions Solve?How Do Transactions Guarantee Exactly-Once?What Problems Aren’t Solved by Transactions?How Do I Use Transactions?Transactional IDs and FencingHow Transactions WorkPerformance of TransactionsSummary
Considerations When Building Data PipelinesTimelinessReliabilityHigh and Varying ThroughputData FormatsTransformationsSecurityFailure HandlingCoupling and AgilityWhen to Use Kafka Connect Versus Producer and ConsumerKafka ConnectRunning Kafka ConnectConnector Example: File Source and File SinkConnector Example: MySQL to ElasticsearchSingle Message TransformationsA Deeper Look at Kafka ConnectAlternatives to Kafka ConnectIngest Frameworks for Other DatastoresGUI-Based ETL ToolsStream Processing FrameworksSummary
Use Cases of Cross-Cluster MirroringMulticluster ArchitecturesSome Realities of Cross-Datacenter CommunicationHub-and-Spoke ArchitectureActive-Active ArchitectureActive-Standby ArchitectureStretch ClustersApache Kafka’s MirrorMakerConfiguring MirrorMakerMulticluster Replication TopologySecuring MirrorMakerDeploying MirrorMaker in ProductionTuning MirrorMakerOther Cross-Cluster Mirroring SolutionsUber uReplicatorLinkedIn BrooklinConfluent Cross-Datacenter Mirroring SolutionsSummary
Locking Down KafkaSecurity ProtocolsAuthenticationSSLSASLReauthenticationSecurity Updates Without DowntimeEncryptionEnd-to-End EncryptionAuthorizationAclAuthorizerCustomizing AuthorizationSecurity ConsiderationsAuditingSecuring ZooKeeperSASLSSLAuthorizationSecuring the PlatformPassword ProtectionSummary
Topic OperationsCreating a New TopicListing All Topics in a ClusterDescribing Topic DetailsAdding PartitionsReducing PartitionsDeleting a TopicConsumer GroupsList and Describe GroupsDelete GroupOffset ManagementDynamic Configuration ChangesOverriding Topic Configuration DefaultsOverriding Client and User Configuration DefaultsOverriding Broker Configuration DefaultsDescribing Configuration OverridesRemoving Configuration OverridesProducing and ConsumingConsole ProducerConsole ConsumerPartition ManagementPreferred Replica ElectionChanging a Partition’s ReplicasDumping Log SegmentsReplica VerificationOther ToolsUnsafe OperationsMoving the Cluster ControllerRemoving Topics to Be DeletedDeleting Topics ManuallySummary
Metric BasicsWhere Are the Metrics?What Metrics Do I Need?Application Health ChecksService-Level ObjectivesService-Level DefinitionsWhat Metrics Make Good SLIs?Using SLOs in AlertingKafka Broker MetricsDiagnosing Cluster ProblemsThe Art of Under-Replicated PartitionsBroker MetricsTopic and Partition MetricsJVM MonitoringOS MonitoringLoggingClient MonitoringProducer MetricsConsumer MetricsQuotasLag MonitoringEnd-to-End MonitoringSummary
What Is Stream Processing?Stream Processing ConceptsTopologyTimeStateStream-Table DualityTime WindowsProcessing GuaranteesStream Processing Design PatternsSingle-Event ProcessingProcessing with Local StateMultiphase Processing/RepartitioningProcessing with External Lookup: Stream-Table JoinTable-Table JoinStreaming JoinOut-of-Sequence EventsReprocessingInteractive QueriesKafka Streams by ExampleWord CountStock Market StatisticsClickStream EnrichmentKafka Streams: Architecture OverviewBuilding a TopologyOptimizing a TopologyTesting a TopologyScaling a TopologySurviving FailuresStream Processing Use CasesHow to Choose a Stream Processing FrameworkSummary
Installing on WindowsUsing Windows Subsystem for LinuxUsing Native JavaInstalling on macOSUsing HomebrewInstalling Manually
Comprehensive PlatformsCluster Deployment and ManagementMonitoring and Data ExplorationClient LibrariesStream Processing

Content preview from Kafka: The Definitive Guide, 2nd Edition

Chapter 2. Installing Kafka

This chapter describes how to get started with the Apache Kafka broker, including how to set up Apache ZooKeeper, which is used by Kafka for storing metadata for the brokers. The chapter will also cover basic configuration options for Kafka deployments, as well as some suggestions for selecting the correct hardware to run the brokers on. Finally, we cover how to install multiple Kafka brokers as part of a single cluster and things you should know when using Kafka in a production environment.

Environment Setup

Before using Apache Kafka, your environment needs to be set up with a few prerequisites to ensure it runs properly. The following sections will guide you through that process.

Choosing an Operating System

Apache Kafka is a Java application and can run on many operating systems. While Kafka is capable of being run on many OSs, including Windows, macOS, Linux, and others, Linux is the recommended OS for the general use case. The installation steps in this chapter will focus on setting up and using Kafka in a Linux environment. For information on installing Kafka on Windows and macOS, see Appendix A.

Installing Java

Prior to installing either ZooKeeper or Kafka, you will need a Java environment set up and functioning. Kafka and ZooKeeper work well with all OpenJDK-based Java implementations, including Oracle JDK. The latest versions of Kafka support both Java 8 and Java 11. The exact version installed can be the version provided by your OS or ...