book

Designing Event-Driven Systems

by Ben Stopford

May 2018

Intermediate to advanced

171 pages

3h 58m

English

O'Reilly Media, Inc.

Read now

Unlock full access

How to Read This BookAcknowledgments
Kafka Is Like REST but Asynchronous?Kafka Is Like a Service Bus?Kafka Is Like a Database?What Is Kafka Really? A Streaming Platform
The Log: An Efficient Structure for Retaining and Distributing MessagesLinear ScalabilitySegregating Load in Multiservice EcosystemsMaintaining Strong Ordering GuaranteesEnsuring Messages Are DurableLoad-Balance Services and Make Them Highly AvailableCompacted TopicsLong-Term Data StorageSecuritySummary
Commands, Events, and QueriesCoupling and Message BrokersIs Loose Coupling Always Good?Essential Data Coupling Is UnavoidableUsing Events for NotificationUsing Events to Provide State TransferWhich Approach to UseThe Event Collaboration PatternRelationship with Stream ProcessingMixing Request- and Event-Driven ProtocolsSummary
Making Services StatefulThe Event-Driven ApproachThe Pure (Stateless) Streaming ApproachThe Stateful Streaming ApproachThe Practicalities of Being StatefulSummary

Event Sourcing, Command Sourcing, and CQRS in a NutshellVersion Control for Your DataMaking Events the Source of TruthCommand Query Responsibility SegregationMaterialized ViewsPolyglot ViewsWhole Fact or Delta?Implementing Event Sourcing and CQRS with KafkaBuild In-Process Views with Tables and State Stores in Kafka StreamsWriting Through a Database into a Kafka Topic with Kafka ConnectWriting Through a State Store to a Kafka Topic in Kafka StreamsUnlocking Legacy Systems with CDCQuery a Read-Optimized View Created in a DatabaseMemory Images/Prepopulated CachesThe Event-Sourced ViewSummary
Encapsulation Isn’t Always Your FriendThe Data DichotomyWhat Happens to Systems as They Evolve?The God Service ProblemThe REST-to-ETL ProblemMake Data on the Outside a First-Class CitizenDon’t Be Afraid to EvolveSummary
A Database Inside OutSummary
If Messaging Remembers, Databases Don’t Have ToTake Only the Data You Need, Nothing MoreRebuilding Event-Sourced ViewsKafka StreamsDatabases and CachesHandling the Impracticalities of Data MovementAutomation and Schema MigrationThe Data Divergence ProblemSummary
Eventual ConsistencyTimelinessCollisions and MergingThe Single Writer PrincipleCommand TopicSingle Writer Per TransitionAtomicity with TransactionsIdentity and Concurrency ControlLimitationsSummary
The Duplicates ProblemUsing the Transactions API to Remove DuplicatesExactly Once Is Both Idempotence and Atomic CommitHow Kafka’s Transactions Work Under the CoversStore State and Send Events AtomicallyDo We Need Transactions? Can We Do All This with Idempotence?What Can’t Transactions Do?Making Use of Transactions in Your ServicesSummary
Using Schemas to Manage the Evolution of Data in TimeHandling Schema Change and Breaking Backward CompatibilityCollaborating over Schema ChangeHandling Unreadable MessagesDeleting DataTriggering Downstream DeletesSegregating Public and Private TopicsSummary
A Simple Email Service Built with Kafka Streams and KSQLWindows, Joins, Tables, and State StoresSummary
An Order Validation EcosystemJoin-Filter-ProcessEvent-Sourced Views in Kafka StreamsCollapsing CQRS with a Blocking ReadScaling Concurrent Operations in Streaming SystemsRekey to JoinRepartitioning and Staged ExecutionWaiting for N EventsReflecting on the DesignA More Holistic Streaming EcosystemSummary

Content preview from Designing Event-Driven Systems

Chapter 4. Beyond Messaging: An Overview of the Kafka Broker

A Kafka cluster is essentially a collection of files, filled with messages, spanning many different machines. Most of Kafka’s code involves tying these various individual logs together, routing messages from producers to consumers reliably, replicating for fault tolerance, and handling failure gracefully. So it is a messaging system, at least of sorts, but it’s quite different from the message brokers that preceded it. Like any technology, it comes with both pros and cons, and these shape the design of the systems we write. This chapter examines the Kafka broker (i.e., the server component) from the context of building business systems. We’ll explore a little about how it works, as well as dipping into the less conventional use cases it supports like data storage, dynamic failover, and bandwidth protection.

Originally built to distribute the datasets created by large social networks, Kafka was predominantly shaped by a need to operate at scale, in the face of failure. Accordingly, its architecture inherits more from storage systems like HDFS, HBase, or Cassandra than it does from traditional messaging systems that implement JMS (Java Message Service) or AMQP (Advanced Message Queuing Protocol).

Like many good outcomes in computer science, this scalability comes largely from simplicity. The underlying abstraction is a partitioned log—essentially a set of append-only files spread over a number of machines—which encourages ...