O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Designing Event-Driven Systems

Book Description

Many forces affect software today: larger datasets, geographical disparities, complex company structures, and the growing need to be fast and nimble in the face of change. Proven approaches such as service-oriented and event-driven architectures are joined by newer techniques such as microservices, reactive architectures, DevOps, and stream processing. Many of these patterns are successful by themselves, but as this practical ebook demonstrates, they provide a more holistic and compelling approach when applied together.

Author Ben Stopford explains how service-based architectures and stream processing tools such as Apache Kafka can help you build business-critical systems. You’ll learn how to apply patterns including Event Sourcing and CQRS, and how to build multi-team systems with microservices and SOA using patterns such as "inside out databases" and "event streams as a source of truth." These approaches provide a unique foundation for how these large, autonomous service ecosystems can communicate and share data.

  • Learn why streaming beats request-response based architectures in complex, contemporary use cases
  • Understand why replayable logs such as Kafka provide a backbone for both service communication and shared datasets
  • Explore how event collaboration and event sourcing patterns increase safety and recoverability with functional, event-driven approaches
  • Build service ecosystems that blend event-driven and request-driven interfaces using a replayable log and Kafka’s Streams API
  • Scale beyond individual teams into larger, department- and company-sized architectures, using event streams as a source of truth

Table of Contents

  1. Foreword
  2. Preface
    1. How to Read This Book
    2. Acknowledgments
  3. I. Setting the Stage
  4. 1. Introduction
  5. 2. The Origins of Streaming
  6. 3. Is Kafka What You Think It Is?
    1. Kafka Is Like REST but Asynchronous?
    2. Kafka Is Like a Service Bus?
    3. Kafka Is Like a Database?
    4. What Is Kafka Really? A Streaming Platform
  7. 4. Beyond Messaging: An Overview of the Kafka Broker
    1. The Log: An Efficient Structure for Retaining and Distributing Messages
    2. Linear Scalability
    3. Segregating Load in Multiservice Ecosystems
    4. Maintaining Strong Ordering Guarantees
    5. Ensuring Messages Are Durable
    6. Load-Balance Services and Make Them Highly Available
    7. Compacted Topics
    8. Long-Term Data Storage
    9. Security
    10. Summary
  8. II. Designing Event-Driven Systems
  9. 5. Events: A Basis for Collaboration
    1. Commands, Events, and Queries
    2. Coupling and Message Brokers
      1. Is Loose Coupling Always Good?
      2. Essential Data Coupling Is Unavoidable
    3. Using Events for Notification
    4. Using Events to Provide State Transfer
    5. Which Approach to Use
    6. The Event Collaboration Pattern
    7. Relationship with Stream Processing
    8. Mixing Request- and Event-Driven Protocols
    9. Summary
  10. 6. Processing Events with Stateful Functions
    1. Making Services Stateful
      1. The Event-Driven Approach
      2. The Pure (Stateless) Streaming Approach
      3. The Stateful Streaming Approach
      4. The Practicalities of Being Stateful
    2. Summary
  11. 7. Event Sourcing, CQRS, and Other Stateful Patterns
    1. Event Sourcing, Command Sourcing, and CQRS in a Nutshell
    2. Version Control for Your Data
    3. Making Events the Source of Truth
    4. Command Query Responsibility Segregation
    5. Materialized Views
    6. Polyglot Views
    7. Whole Fact or Delta?
    8. Implementing Event Sourcing and CQRS with Kafka
      1. Build In-Process Views with Tables and State Stores in Kafka Streams
      2. Writing Through a Database into a Kafka Topic with Kafka Connect
      3. Writing Through a State Store to a Kafka Topic in Kafka Streams
      4. Unlocking Legacy Systems with CDC
      5. Query a Read-Optimized View Created in a Database
      6. Memory Images/Prepopulated Caches
      7. The Event-Sourced View
    9. Summary
  12. III. Rethinking Architecture at Company Scales
  13. 8. Sharing Data and Services Across an Organization
    1. Encapsulation Isn’t Always Your Friend
    2. The Data Dichotomy
    3. What Happens to Systems as They Evolve?
      1. The God Service Problem
      2. The REST-to-ETL Problem
    4. Make Data on the Outside a First-Class Citizen
    5. Don’t Be Afraid to Evolve
    6. Summary
  14. 9. Event Streams as a Shared Source of Truth
    1. A Database Inside Out
    2. Summary
  15. 10. Lean Data
    1. If Messaging Remembers, Databases Don’t Have To
    2. Take Only the Data You Need, Nothing More
    3. Rebuilding Event-Sourced Views
      1. Kafka Streams
      2. Databases and Caches
      3. Handling the Impracticalities of Data Movement
    4. Automation and Schema Migration
      1. The Data Divergence Problem
    5. Summary
  16. IV. Consistency, Concurrency, and Evolution
  17. 11. Consistency and Concurrency in Event-Driven Systems
    1. Eventual Consistency
      1. Timeliness
      2. Collisions and Merging
    2. The Single Writer Principle
      1. Command Topic
      2. Single Writer Per Transition
    3. Atomicity with Transactions
    4. Identity and Concurrency Control
    5. Limitations
    6. Summary
  18. 12. Transactions, but Not as We Know Them
    1. The Duplicates Problem
    2. Using the Transactions API to Remove Duplicates
    3. Exactly Once Is Both Idempotence and Atomic Commit
    4. How Kafka’s Transactions Work Under the Covers
    5. Store State and Send Events Atomically
    6. Do We Need Transactions? Can We Do All This with Idempotence?
    7. What Can’t Transactions Do?
    8. Making Use of Transactions in Your Services
    9. Summary
  19. 13. Evolving Schemas and Data over Time
    1. Using Schemas to Manage the Evolution of Data in Time
    2. Handling Schema Change and Breaking Backward Compatibility
    3. Collaborating over Schema Change
    4. Handling Unreadable Messages
    5. Deleting Data
      1. Triggering Downstream Deletes
    6. Segregating Public and Private Topics
    7. Summary
  20. V. Implementing Streaming Services with Kafka
  21. 14. Kafka Streams and KSQL
    1. A Simple Email Service Built with Kafka Streams and KSQL
    2. Windows, Joins, Tables, and State Stores
    3. Summary
  22. 15. Building Streaming Services
    1. An Order Validation Ecosystem
    2. Join-Filter-Process
    3. Event-Sourced Views in Kafka Streams
    4. Collapsing CQRS with a Blocking Read
    5. Scaling Concurrent Operations in Streaming Systems
    6. Rekey to Join
    7. Repartitioning and Staged Execution
    8. Waiting for N Events
    9. Reflecting on the Design
    10. A More Holistic Streaming Ecosystem
    11. Summary