book

Kafka Streams in Action, Second Edition

Name: Kafka Streams in Action, Second Edition
Author: Bill Bejeck
ISBN: 9781617298684

by Bill Bejeck

May 2024

Intermediate to advanced

504 pages

15h

English

Manning Publications

Read now

Unlock full access

copyright
contents
Praise for the first edition
Kafka Streams in Action
foreword
preface
acknowledgments
about this book
about the author
about the cover illustration

Part 1
1 Welcome to the Kafka event streaming platform
1.1 Event streaming1.2 What is an event?1.3 An event stream example1.4 Introducing the Apache Kafka event streaming platform1.4.1 Kafka brokers1.4.2 Schema Registry1.4.3 Producer and consumer clients1.4.4 Kafka Connect1.4.5 Kafka Streams1.4.6 ksqlDB1.5 A concrete example of applying the Kafka event streaming platform
2 Kafka brokers
2.1 Introducing Kafka brokers2.2 Produce requests2.3 Fetch requests2.4 Topics and partitions2.4.1 Offsets2.4.2 Determining the correct number of partitions2.5 Sending your first messages2.5.1 Creating a topic2.5.2 Producing records on the command line2.5.3 Consuming records from the command line2.5.4 Partitions in action2.6 Segments2.6.1 Data retention2.6.2 Compacted topics2.6.3 Topic partition directory contents2.7 Tiered storage2.8 Cluster metadata2.9 Leaders and followers2.9.1 Replication2.10 Checking for a healthy broker2.10.1 Request handler idle percentage2.10.2 Network handler idle percentage2.10.3 Underreplicated partitions
Part 2
3 Schema Registry
3.1 Objects3.2 What is a schema, and why do you need one?3.2.1 What is Schema Registry?3.2.2 Getting Schema Registry3.2.3 Architecture3.2.4 Communication: Using Schema Registry’s REST API3.2.5 Registering a schema3.2.6 Plugins and serialization platform tools3.2.7 Uploading a schema file3.2.8 Generating code from schemas3.2.9 End-to-end example3.3 Subject name strategies3.3.1 TopicNameStrategy3.3.2 RecordNameStrategy3.3.3 TopicRecordNameStrategy3.4 Schema compatibility3.4.1 Backward compatibility3.4.2 Forward compatibility3.4.3 Full compatibility3.4.4 No compatibility3.5 Schema references3.6 Schema references and multiple events per topic3.7 Schema Registry (de)serializers3.7.1 Avroserializers and deserializers3.7.2 Protobuf3.7.3 JSON Schema3.8 Serialization without Schema Registry
4 Kafka clients
4.1 Introducing Kafka clients4.2 Producing records with the KafkaProducer4.2.1 Producer configurations4.2.2 Kafka delivery semantics4.2.3 Partition assignment4.2.4 Writing a custom partitioner4.2.5 Specifying a custom partitioner4.2.6 Timestamps4.3 Consuming records with the KafkaConsumer4.3.1 The poll interval4.3.2 The group id configuration4.3.3 Applying partition assignment strategies4.3.4 Static membership4.3.5 Committing offsets4.4 Exactly-once delivery in Kafka4.4.1 The idempotent producer4.4.2 Transactional producer4.4.3 Consumers in transactions4.4.4 Producers and consumers within a transaction4.5 Using the Admin API for programmatic topic management4.6 Handling multiple event types in a single topic4.6.1 Producing multiple event types4.6.2 Consuming multiple event types
5 Kafka Connect
5.1 An introduction to Kafka Connect5.2 Integrating external applications into Kafka5.3 Getting started with Kafka Connect5.4 Applying Single Message Transforms5.5 Adding a sink connector5.6 Building and deploying your own connector5.6.1 Implementing a connector5.6.2 Making your connector dynamic with a monitoring thread5.6.3 Creating a custom transformation
Part 3
6 Developing Kafka Streams
6.1 A look at Kafka Streams6.2 Kafka Streams DSL6.3 Hello World for Kafka Streams6.3.1 Creating the topology for the Yelling app6.3.2 Kafka Streams configuration6.3.3 Serde creation6.4 Masking credit card numbers and tracking purchase rewards in a retail sales setting6.4.1 Building the source node and the masking processor6.4.2 Adding the purchase-patterns processor6.4.3 Building the rewards processor6.4.4 Using Serdes to encapsulate serializers and deserializers in Kafka Streams6.4.5 Kafka Streams and Schema Registry6.5 Interactive development6.6 Choosing which events to process6.6.1 Filtering purchases6.6.2 Splitting/branching the stream6.6.3 Naming topology nodes6.6.4 Dynamic routing of messages
7 Streams and state
7.1 Stateful vs. stateless7.2 Adding stateful operations to Kafka Streams7.2.1 Group-by details7.2.2 Aggregation vs. reducing7.2.3 Repartitioning the data7.2.4 Proactive repartitioning7.2.5 Repartitioning to increase the number of tasks7.2.6 Using Kafka Streams optimizations7.3 Stream-stream joins7.3.1 Implementing a stream-stream join7.3.2 Join internals7.3.3 ValueJoiner7.3.4 JoinWindows7.3.5 Co-partitioning7.3.6 StreamJoined7.3.7 Other join options7.3.8 Outer joins7.3.9 Left-outer join7.4 State stores in Kafka Streams7.4.1 Changelog topics restoring state stores7.4.2 Standby tasks7.4.3 Assigning state stores in Kafka Streams7.4.4 State stores’ location on the filesystem7.4.5 Naming stateful operations7.4.6 Specifying a store type7.4.7 Configuring changelog topics
8 The KTable API
8.1 KTable: The update stream8.1.1 Updates to records or the changelog8.1.2 KStream and KTable API in action8.2 KTables are stateful8.3 The KTable API8.4 KTable aggregations8.5 GlobalKTable8.6 Table joins8.6.1 Stream–table join details8.6.2 Versioned KTables8.6.3 Stream–global table join details8.6.4 Table–table join details
9 Windowing and timestamps
9.1 Understanding the role of windows and the different types9.1.1 Hopping windows9.1.2 Tumbling windows9.1.3 Session windows9.1.4 Sliding windows9.1.5 Window time alignment9.1.6 Retrieving window results for analysis9.2 Handling out order data with grace—literally9.3 Final windowed results9.3.1 Strict buffering9.3.2 Eager buffering9.4 Timestamps in Kafka Streams9.5 The TimestampExtractor9.5.1 WallclockTimestampExtractorSystem .currentTimeMillis() method9.5.2 Custom TimestampExtractor9.5.3 Specifying a TimestampExtractor9.6 Stream time
10 The Processor API
10.1 Working with sources, processors, and sinks to create a topology10.1.1 Adding a source node10.1.2 Adding a processor node10.1.3 Adding a sink node10.2 Digging deeper into the Processor API with a stock analysis processor10.2.1 The stock-performance processor application10.2.2 Punctuation semantics10.2.3 The process() method10.2.4 The punctuator execution10.3 Data-driven aggregation10.4 Integrating the Processor API and the Kafka Streams API
11 ksqlDB
11.1 Understanding ksqlDB11.2 More about streaming queries11.3 Persistent vs. push vs. pull queries11.4 Creating Streams and Tables11.5 Schema Registry integration11.6 ksqlDB advanced features
12 Spring Kafka
12.1 Introducing Spring12.2 Using Spring to build Kafka-enabled applications12.2.1 Spring Kafka application components12.2.2 Enhanced application requirements12.3 Spring Kafka Streams
13 Kafka Streams Interactive Queries
13.1 Kafka Streams and information sharing13.2 Learning about Interactive Queries13.2.1 Building an Interactive Queries app with Spring Boot
14 Testing
14.1 Understanding the difference between unit and integration testing14.1.1 Testing Kafka producers and consumers14.1.2 Creating tests for Kafka Streams operators14.1.3 Writing tests for a Kafka Streams topology14.1.4 Testing more complex Kafka Streams applications14.1.5 Developing effective integration tests
appendix A Schema compatibility workshop
A.1 Backward compatibilityA.2 Forward compatibilityA.3 Full compatibility
appendix B Confluent resources
B.1 Confluent CloudB.2 Confluent command-line interfaceB.3 Confluent local
appendix C Working with Avro, Protobuf, and JSON Schema
C.1 Apache AvroC.1.1 Default and aliasC.1.2 UnionC.2 Protocol BuffersC.2.1 Complex messagesC.2.2 ImportingC.2.3 Oneof typeC.2.4 Code generationC.2.5 Specific and dynamic typesC.3 JSON SchemaC.3.1 Nested objectsC.3.2 JSON referencesC.3.3 JSON Schema Registry schema referencesC.3.4 JSON Schema code generationC.3.5 Specific and generic types
appendix D Understanding Kafka Streams architecture
D.1 High-level viewD.2 Consumer and producer clients in Kafka StreamsD.3 Assigning, distributing, and processing eventsD.4 Threads in Kafka Streams: StreamThreadD.5 Processing records

Content preview from Kafka Streams in Action, Second Edition

appendix C Working with Avro, Protobuf, and JSON Schema

C.1 Apache Avro

An Avro schema is composed of two main types: primitive and complex. The supported primitive types are null, boolean, int, long, float, double, bytes, and string. Complex types are records, enums, arrays, maps, unions, and fixed. For a complete description of primitive and complex types, see the Avro documentation at http://mng.bz/Y7Oo and http://mng.bz/GZ2M, respectively.

In this book, you’ll mostly work with the complex type of record as this corresponds to an object. You may also encounter a few examples using primitive values, but for the most part, we’ll stick with records. You’ll also use the union type, especially when working with the RecordNameStrategy

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617298684Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Kafka Streams in Action, Second Edition

by Bill Bejeck

appendix C Working with Avro, Protobuf, and JSON Schema

C.1 Apache Avro

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.