book

Building Event-Driven Microservices, 2nd Edition

Name: Building Event-Driven Microservices, 2nd Edition
Author: Adam Bellemare
ISBN: 9798341622197

by Adam Bellemare

September 2025

Intermediate to advanced

484 pages

13h 27m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Foreword
Preface
Conventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgements
I. Introduction to Event-Driven Microservices
1. Why Event-Driven Microservices
What Are Event-Driven Microservices?Introduction to Domain-Driven Design and Bounded ContextsMicroservices, Boundaries, and Communication StructuresBusiness Communication StructuresImplementation Communication StructuresData Communication StructuresConway’s Law and Communication StructuresCommunication Structures in Traditional ComputingOption 1: Make a New ServiceOption 2: Add It to the Existing ServicePros and Cons of Each OptionThe Team Scenario, ContinuedEvent-Driven Data CommunicationExample Team Using Event-Driven MicroservicesRequest-Response MicroservicesDrawbacks of Request-Response MicroservicesBenefits of Request-Response MicroservicesSummary
2. Fundamentals of Events and Event Streams
What’s an Event?What’s an Event Stream?Ephemeral MessagingQueuingQueues via Modern Queue BrokersQueues via Apache KafkaThe Structure of an EventUnkeyed EventsKeyed EventsEntity EventsRepartitioning Event StreamsCopartitioning Event StreamsAggregating State from Keyed EventsMaterializing State from Entity EventsDeleting Events and Event-Stream CompactionThe Kappa ArchitectureThe Lambda ArchitectureEvent Data Definitions and SchemasPowering Microservices with the Event BrokerSelecting an Event BrokerSupport ToolingHosted ServicesClient Libraries and Processing FrameworksCommunity SupportIndefinite and Tiered StorageSummary
3. Fundamentals of Event-Driven Microservices
The Basics of Event-Driven MicroservicesThe Basic Producer/ConsumerThe Stream-Processing Event-Driven MicroserviceThe Streaming SQL QueryThe Legacy ApplicationTopologies and Event-Driven MicroservicesMicroservice TopologyBusiness TopologyEvent-Driven Microservice ResponsibilitiesHow Small Should a Microservice Be?Managing Microservices at ScalePutting Microservices into ContainersPutting Microservices into Virtual MachinesManaging Containers and Virtual MachinesPaying the Microservice TaxService ContractsSummary
II. Events and Event Streams
4. Schemas and Data Contracts
A Brief Introduction to Serialization and DeserializationWhat Is a Schema?What Are Your Schema Technology Options?Google’s Protocol Buffers, aka ProtobufApache AvroJSON SchemaSchema Evolution: Changing Your Schemas Through TimeWhat Is a Data Contract?Negotiating a Breaking Schema ChangeStep 1: Design the New SchemaStep 2: Consult Your Existing Consumers and Gain ApprovalStep 3: Plan the Release, Migration, and DeprecationStep 4: Execute the ReleaseThe Role of the Schema RegistryField-Level EncryptionIntegration with Data CatalogsSummary
5. Designing Events
Introduction to Event TypesState Events and Event-Carried State TransferCurrent State EventsBefore/After State EventsDelta EventsDelta Events for Event SourcingThe Problems with Delta EventsWhere Do I Use Delta and State Events?Hybrid Events: State with a Bit of DeltaMeasurement EventsMeasurement Events Enable AggregationsMeasurement Event Sources May Be LossyMeasurement Events Can Power Time-Sensitive ApplicationsNotification EventsEvent Design for Data Privacy and the Right to Be ForgottenSummary
6. Integrating Event-Driven Architectures with Existing Systems
What Is Data Liberation?The Dual Write AntipatternData Liberation PatternsData Liberation FrameworksLiberating Data Using Change-Data CaptureSnapshotting the Initial Data Set StateBenefits of Change-Data Capture Using Data Store LogsDrawbacks of Change-Data Capture Using Data Store LogsLiberating Data with a Polling QueryIncremental UpdatingBenefits of Query-Based UpdatingDrawbacks of Query-Based UpdatingLiberating Data Using Transactional Outbox TablesPerformance ConsiderationsIsolating Internal Data ModelsEnsuring Schema CompatibilityCapturing Change-Data Using TriggersMaking Data Definition Changes to Data Sets Under CaptureCompromises for Data LiberationSinking Event Data to Data StoresThe Impacts of Sinking and Sourcing on a BusinessSummary

7. Denormalization and Eventification
Eventification at the Transactional OutboxEventification in a Dedicated ServiceWhat Should Go in the Event? And What Should Stay Out?Slowly Changing DimensionsType 1: Overwrite with the New ValueType 2: Append the New ValueSummary
8. Stateful Event-Driven Microservices
Event-Stream Partitions and Consumer AssignmentsRepartitioning Event StreamsCopartitioning Event StreamsAssigning Partitions Within a Consumer GroupSelecting a State Store for Your MicroserviceMaterializing State to an Internal State StoreUsing a Changelog Event Stream as State RecoveryMaterializing Global StateAdvantages of Internal State StoresDisadvantages of Internal State StoresScaling and Recovery of Internal StateMaterializing State to an External State StoreAdvantages of External StateDrawbacks of External StateScaling and Recovery with External State StoresRebuilding the External State StoreTransactions and Effectively Once ProcessingExample: Effectively Once Processing for an Inventory Accounting ServiceEffectively Once Processing with Client-Broker TransactionsEffectively Once Processing Without Client-Broker TransactionsSummary
9. Deterministic Stream Processing
Determinism with Event-Driven WorkflowsTimestampsSynchronizing Distributed TimestampsEvent Scheduling and Deterministic ProcessingCustom Event SchedulersProcessing Based on Event Time, Processing Time, and Ingestion TimeTimestamp Extraction by the ConsumerRequest-Response Calls to External SystemsWatermarksWatermarks in Parallel ProcessingStream TimeOut-of-Order and Late-Arriving EventsLate Events with Watermarks and Stream TimeCauses and Impacts of Out-of-Order EventsTime-Sensitive Functions and WindowingHandling Late EventsReprocessing Versus Processing in Near–Real TimeIntermittent Failures and Late EventsProducer/Event Broker Connectivity IssuesSummary and Further Reading
10. Building Workflows with Microservices
The Choreography PatternModifying a Choreographed WorkflowMonitoring a Choreographed WorkflowThe Orchestration PatternA Simple Event-Driven Orchestration ExampleA Simple Request-Response Orchestration ExampleModifying an Orchestration WorkflowMonitoring an Orchestration WorkflowComparing Request-Response and Event-Driven OrchestrationDistributed Transactions and the Saga PatternSagas via ChoreographySagas via OrchestrationThe Compensation Workflow PatternImplementing Orchestration via a Durable Execution EngineEvent Sourcing via the Durable Append-Only LogFurther Considerations of Durable ExecutionSummary
III. Event-Driven Microservices Frameworks
11. Basic Producer and Consumer Microservices
Where Do BPCs Work Well?Integration with Existing Systems Using the Sidecar PatternStateful Business Logic That Isn’t Reliant Upon Event OrderWhen the Data Store Does Much of the WorkIndependent Scaling of the Processing and Data StoreSummary
12. Heavyweight Framework Microservices
A Brief History of Heavyweight FrameworksExample: Session Windowing of Clicks and ViewsThe Inner Workings of Heavyweight FrameworksBenefits and LimitationsCluster Setup Options and Execution ModesUse a Hosted ServiceBuild and Run Your Own ClusterApplication Submission ModesDriver ModeCluster ModeHandling State and Using CheckpointsScaling Applications and Handling Event-Stream PartitionsScaling an Application by Restarting ItScaling an Application While It Is RunningAutoscaling ApplicationsRecovering from FailuresMultitenancy ConsiderationsLanguages and SyntaxChoosing a FrameworkSummary
13. Lightweight Framework Microservices
Example: Joining Products with Brand Data on a Foreign KeyUnder the Hood of Lightweight FrameworksEvent Shuffling and RepartitioningHandling State and Using ChangelogsScaling and Recovering from FailuresState AssignmentState Replication and Hot ReplicasSummary
14. Streaming SQL
The Basics of the Continuous QueryTurning Streams into Tables for Streaming SQL QueriesExample: Enriching Data Using Flink SQLExecuting Streaming SQL CodeExecuting SQL Code from the Command-Line InterfaceExecuting SQL Code from Within the ApplicationExecuting SQL Code from a NotebookExecuting SQL Code Via a GatewaySQL as a SidecarUser-Defined Function CallsSummary
15. Microservices Using Functions as a Service
Why Would I Use Functions as a Service?Building Microservices with FunctionsCold Starts and Warm StartsTermination and ShutdownChoosing a FaaS ProviderStarting Functions with TriggersTriggering Based on New Events: The Event-Stream ListenerTriggering Based on Consumer Group LagTriggering on a ScheduleTriggering Using WebhooksTriggering on Resource EventsScaling Your FaaS SolutionsMaintaining StateFunctions Calling Other FunctionsAsynchronous Event-Driven Communication PatternDirect-Call PatternHandling FaaS Function FailuresDurable Function OrchestratorsEncoding the Orchestrator in a Proprietary DocumentEncoding the Orchestrator Within the Source CodeSummary
IV. Consistency, Bad Data, and Supportive Tooling
16. Eventual Consistency
Converging on Consistency, One Event at a TimeStrategies for Dealing with Eventual ConsistencyUse State Event TypesExpose Eventual Consistency in the Server ResponseUse Event-Driven Communication Instead of Request-ResponsePrevent Failures to Avoid InconsistencySummary
17. Integrating Event-Driven and Request-Response Microservices
Turning Requests into EventsTurning Requests into Events Using a REST ProxyIntegrating with Third-Party Request-Response APIsServing Data Using a Request-Response APIServing Requests with Internal State StoresServing Requests with External State StoresHandling Requests Within an Event-Driven WorkflowProcessing Events for User InterfacesExample: Newspaper Publishing Workflow (Approval Pattern)Example: Separating the Editor and Advertiser Approval ServicesMicro Frontends for Request-Response ApplicationsExample: Experience Search and Review ApplicationThe Benefits of Micro FrontendsThe Drawbacks of Micro FrontendsSummary
18. Handling Bad Data in Event Streams
The Main Types of Bad Data in Event StreamsType 1: Corrupted DataType 2: Event Has No SchemaType 3: Event Has an Invalid SchemaType 4: Incompatible Schema EvolutionType 5: Logically Invalid Value in a FieldType 6: Logically Valid Value but Semantically IncorrectType 7: Missing EventsType 8: Events That Should Not Have Been ProducedPreventing Bad Data with Schemas, Validation, and TestsPreventing Bad Data Types 1–5 with Schemas and Schema EvolutionData Quality Rules: Handling Type 6: (Logically Valid But Semantically Incorrect)Testing: Handling Types 7 (Missing Data) and 8 (Data That Should Not Have Been Produced)The Role of Event Design in Fixing Bad DataFix It Once and Fix It Right with State EventsBuild Forward: Undo Bad Deltas with New DeltasThe Last Resort: Rewind, Rebuild, and RetryRewind, Rebuild, and Retry from an External SourceRewind, Rebuild, and Retry with the Topic as the SourceSummary
19. Supportive Tooling
Choosing Your Infrastructure: Build Versus BuyInfrastructure as Code for Clusters and ServicesIdentity and Access ManagementMicroservice-to-Team Assignment SystemEvent-Stream Creation and ModificationEvent Stream and Microservice Metadata TaggingQuotasSchema RegistryManaging Event-Stream PermissionsSchema Creation and Modification NotificationsConsumer Offset ManagementState Management and Application ResetConsumer Offset Lag MonitoringStreamline the Microservice Creation ProcessContainer Management ControlsMulticluster Deployments and Event-Stream ReplicationDependency Tracking and Topology VisualizationSummary
20. Testing Event-Driven Microservices
General Testing PrinciplesUnit-Testing Microservice FunctionsUnit Testing Stateless FunctionsUnit Testing Stateful FunctionsTesting the Full Event-Driven TopologyTesting Schema Evolution and CompatibilityIntegration Testing of Event-Driven MicroservicesLocal Integration TestingCreate a Testing Environment Within the Runtime of Your Test CodeCreating a Testing Environment with ContainersIntegrating Hosted, Managed, and SaaS ServicesTesting Using Fully Remote Cloud ServicesProgrammatically Create a Temporary Single-Tenant Testing EnvironmentTesting Using a Durable Multitenant EnvironmentTesting Using the Production EnvironmentChoosing Your Full-Remote Integration Testing StrategySummary
21. Deploying Event-Driven Microservices
Principles of Microservice DeploymentArchitectural Components of Microservice DeploymentContinuous Integration, Delivery, and Deployment SystemsContainer Management Systems and Commodity HardwareThe Basic Full-Stop Deployment PatternThe Rolling Update PatternThe Breaking Schema Change PatternVariant 1: Write the Breaking Events to the Same Old StreamVariant 2: Write the Breaking Events to a New StreamVariant 3: Write Both the Old and New Events in ParallelReprocessing Historical Data for Breaking Schema ChangesThe Blue-Green Deployment PatternSummary
22. Conclusion
The Data Communication StructureBusiness Domains and Bounded ContextsTooling, Infrastructure, and the Microservice TaxEvent DesignSchematized EventsData Liberation and the Single Source of TruthMicroservicesMicroservice Implementation OptionsTestingDeployingFinal Words
Index
About the Author

Content preview from Building Event-Driven Microservices, 2nd Edition

About the Author

Adam Bellemare is a principal technologist of the Technology Strategy Group at Confluent. Previously, he has worked as a staff engineer on data streaming platforms at Shopify and Flipp from 2014 to 2021. He started his work in event-driven systems at Blackberry in 2010, using custom-built streaming data pipelines for analyzing and reporting on cell-phone failures.

His expertise includes DevOps (Kafka, Spark, Mesos, Kubernetes, Solr, Elasticsearch, HBase, and Zookeeper clusters, programmatic building, scaling, monitoring); technical leadership (helping businesses organize their data communication layer, integrate existing systems, develop new systems, and focus on delivering products); software development (building event-driven microservices in Java and Scala using Kafka Streams, Flink, and Spark); and data engineering (reshaping the way that behavioral data is collected from user devices and shared within the organization).

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9798341622180Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Building Event-Driven Microservices, 2nd Edition

by Adam Bellemare

About the Author

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.