Designing Event-Driven Systems

Table of contents

1. Foreword
2. Preface
   1. How to Read This Book
   2. Acknowledgments
3. I. Setting the Stage
4. 1. Introduction
5. 2. The Origins of Streaming
6. 3. Is Kafka What You Think It Is?
   1. Kafka Is Like REST but Asynchronous?
   2. Kafka Is Like a Service Bus?
   3. Kafka Is Like a Database?
   4. What Is Kafka Really? A Streaming Platform
7. 4. Beyond Messaging: An Overview of the Kafka Broker
   1. The Log: An Efficient Structure for Retaining and Distributing Messages
   2. Linear Scalability
   3. Segregating Load in Multiservice Ecosystems
   4. Maintaining Strong Ordering Guarantees
   5. Ensuring Messages Are Durable
   6. Load-Balance Services and Make Them Highly Available
   7. Compacted Topics
   8. Long-Term Data Storage
   9. Security
   10. Summary
8. II. Designing Event-Driven Systems
9. 5. Events: A Basis for Collaboration
   1. Commands, Events, and Queries
   2. Coupling and Message Brokers
      1. Is Loose Coupling Always Good?
      2. Essential Data Coupling Is Unavoidable
   3. Using Events for Notification
   4. Using Events to Provide State Transfer
   5. Which Approach to Use
   6. The Event Collaboration Pattern
   7. Relationship with Stream Processing
   8. Mixing Request- and Event-Driven Protocols
   9. Summary
10. 6. Processing Events with Stateful Functions
    1. Making Services Stateful
       1. The Event-Driven Approach
       2. The Pure (Stateless) Streaming Approach
       3. The Stateful Streaming Approach
       4. The Practicalities of Being Stateful
    2. Summary
11. 7. Event Sourcing, CQRS, and Other Stateful Patterns
    1. Event Sourcing, Command Sourcing, and CQRS in a Nutshell
    2. Version Control for Your Data
    3. Making Events the Source of Truth
    4. Command Query Responsibility Segregation
    5. Materialized Views
    6. Polyglot Views
    7. Whole Fact or Delta?
    8. Implementing Event Sourcing and CQRS with Kafka
       1. Build In-Process Views with Tables and State Stores in Kafka Streams
       2. Writing Through a Database into a Kafka Topic with Kafka Connect
       3. Writing Through a State Store to a Kafka Topic in Kafka Streams
       4. Unlocking Legacy Systems with CDC
       5. Query a Read-Optimized View Created in a Database
       6. Memory Images/Prepopulated Caches
       7. The Event-Sourced View
    9. Summary
12. III. Rethinking Architecture at Company Scales
13. 8. Sharing Data and Services Across an Organization
    1. Encapsulation Isn’t Always Your Friend
    2. The Data Dichotomy
    3. What Happens to Systems as They Evolve?
       1. The God Service Problem
       2. The REST-to-ETL Problem
    4. Make Data on the Outside a First-Class Citizen
    5. Don’t Be Afraid to Evolve
    6. Summary
14. 9. Event Streams as a Shared Source of Truth
    1. A Database Inside Out
    2. Summary
15. 10. Lean Data
    1. If Messaging Remembers, Databases Don’t Have To
    2. Take Only the Data You Need, Nothing More
    3. Rebuilding Event-Sourced Views
       1. Kafka Streams
       2. Databases and Caches
       3. Handling the Impracticalities of Data Movement
    4. Automation and Schema Migration
       1. The Data Divergence Problem
    5. Summary
16. IV. Consistency, Concurrency, and Evolution
17. 11. Consistency and Concurrency in Event-Driven Systems
    1. Eventual Consistency
       1. Timeliness
       2. Collisions and Merging
    2. The Single Writer Principle
       1. Command Topic
       2. Single Writer Per Transition
    3. Atomicity with Transactions
    4. Identity and Concurrency Control
    5. Limitations
    6. Summary
18. 12. Transactions, but Not as We Know Them
    1. The Duplicates Problem
    2. Using the Transactions API to Remove Duplicates
    3. Exactly Once Is Both Idempotence and Atomic Commit
    4. How Kafka’s Transactions Work Under the Covers
    5. Store State and Send Events Atomically
    6. Do We Need Transactions? Can We Do All This with Idempotence?
    7. What Can’t Transactions Do?
    8. Making Use of Transactions in Your Services
    9. Summary
19. 13. Evolving Schemas and Data over Time
    1. Using Schemas to Manage the Evolution of Data in Time
    2. Handling Schema Change and Breaking Backward Compatibility
    3. Collaborating over Schema Change
    4. Handling Unreadable Messages
    5. Deleting Data
       1. Triggering Downstream Deletes
    6. Segregating Public and Private Topics
    7. Summary
20. V. Implementing Streaming Services with Kafka
21. 14. Kafka Streams and KSQL
    1. A Simple Email Service Built with Kafka Streams and KSQL
    2. Windows, Joins, Tables, and State Stores
    3. Summary
22. 15. Building Streaming Services
    1. An Order Validation Ecosystem
    2. Join-Filter-Process
    3. Event-Sourced Views in Kafka Streams
    4. Collapsing CQRS with a Blocking Read
    5. Scaling Concurrent Operations in Streaming Systems
    6. Rekey to Join
    7. Repartitioning and Staged Execution
    8. Waiting for N Events
    9. Reflecting on the Design
    10. A More Holistic Streaming Ecosystem
    11. Summary