book

Building Event-Driven Microservices

by Adam Bellemare

July 2020

Beginner to intermediate

321 pages

9h 10m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Conventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
What Are Event-Driven Microservices?Introduction to Domain-Driven Design and Bounded ContextsLeveraging Domain Models and Bounded ContextsAligning Bounded Contexts with Business RequirementsCommunication StructuresBusiness Communication StructuresImplementation Communication StructuresData Communication StructuresConway’s Law and Communication StructuresCommunication Structures in Traditional ComputingOption 1: Make a New ServiceOption 2: Add It to the Existing ServicePros and Cons of Each OptionThe Team Scenario, ContinuedConflicting PressuresEvent-Driven Communication StructuresEvents Are the Basis of CommunicationEvent Streams Provide the Single Source of TruthConsumers Perform Their Own Modeling and QueryingData Communication Is Improved Across the OrganizationAccessible Data Supports Business Communication ChangesAsynchronous Event-Driven MicroservicesExample Team Using Event-Driven MicroservicesSynchronous MicroservicesDrawbacks of Synchronous MicroservicesBenefits of Synchronous MicroservicesSummary
Building TopologiesMicroservice TopologyBusiness TopologyThe Contents of an EventThe Structure of an EventUnkeyed EventEntity EventKeyed EventMaterializing State from Entity EventsEvent Data Definitions and SchemasMicroservice Single Writer PrinciplePowering Microservices with the Event BrokerEvent Storage and ServingAdditional Factors to ConsiderEvent Brokers Versus Message BrokersConsuming from the Immutable LogProviding a Single Source of TruthManaging Microservices at ScalePutting Microservices into ContainersPutting Microservices into Virtual MachinesManaging Containers and Virtual MachinesPaying the Microservice TaxSummary
Event-Driven Data ContractsUsing Explicit Schemas as ContractsSchema Definition CommentsFull-Featured Schema EvolutionCode Generator SupportBreaking Schema ChangesSelecting an Event FormatDesigning EventsTell the Truth, the Whole Truth, and Nothing but the TruthUse a Singular Event Definition per StreamUse the Narrowest Data TypesKeep Events Single-PurposeMinimize the Size of EventsInvolve Prospective Consumers in the Event DesignAvoid Events as Semaphores or SignalsSummary
What Is Data Liberation?Compromises for Data LiberationConverting Liberated Data to EventsData Liberation PatternsData Liberation FrameworksLiberating Data by QueryBulk LoadingIncremental Timestamp LoadingAutoincrementing ID LoadingCustom QueryingIncremental UpdatingBenefits of Query-Based UpdatingDrawbacks of Query-Based UpdatingLiberating Data Using Change-Data Capture LogsBenefits of Using Data Store LogsDrawbacks of Using Data Base LogsLiberating Data Using Outbox TablesPerformance ConsiderationsIsolating Internal Data ModelsEnsuring Schema CompatibilityCapturing Change-Data Using TriggersMaking Data Definition Changes to Data Sets Under CaptureHandling After-the-Fact Data Definition Changes for the Query and CDC Log PatternsHandling Data Definition Changes for Change-Data Table Capture PatternsSinking Event Data to Data StoresThe Impacts of Sinking and Sourcing on a BusinessSummary
Composing Stateless TopologiesTransformationsBranching and Merging StreamsRepartitioning Event StreamsExample: Repartitioning an Event StreamCopartitioning Event StreamsExample: Copartitioning an Event StreamAssigning Partitions to a Consumer InstanceAssigning Partitions with the Partition AssignorAssigning Copartitioned PartitionsPartition Assignment StrategiesRecovering from Stateless Processing Instance FailuresSummary
Determinism with Event-Driven WorkflowsTimestampsSynchronizing Distributed TimestampsProcessing with Timestamped EventsEvent Scheduling and Deterministic ProcessingCustom Event SchedulersProcessing Based on Event Time, Processing Time, and Ingestion TimeTimestamp Extraction by the ConsumerRequest-Response Calls to External SystemsWatermarksWatermarks in Parallel ProcessingStream TimeStream Time in Parallel ProcessingOut-of-Order and Late-Arriving EventsLate Events with Watermarks and Stream TimeCauses and Impacts of Out-of-Order EventsTime-Sensitive Functions and WindowingHandling Late EventsReprocessing Versus Processing in Near-Real TimeIntermittent Failures and Late EventsProducer/Event Broker Connectivity IssuesSummary and Further Reading
State Stores and Materializing State from an Event StreamRecording State to a Changelog Event StreamMaterializing State to an Internal State StoreMaterializing Global StateAdvantages of Using Internal StateDisadvantages of Using Internal StateScaling and Recovery of Internal StateMaterializing State to an External State StoreAdvantages of External StateDrawbacks of External StateScaling and Recovery with External State StoresRebuilding Versus Migrating State StoresRebuildingMigratingTransactions and Effectively Once ProcessingExample: Stock Accounting ServiceEffectively Once Processing with Client-Broker TransactionsEffectively Once Processing Without Client-Broker TransactionsSummary
The Choreography PatternA Simple Event-Driven Choreography ExampleCreating and Modifying a Choreographed WorkflowMonitoring a Choreographed WorkflowThe Orchestration PatternA Simple Event-Driven Orchestration ExampleA Simple Direct-Call Orchestration ExampleComparing Direct-Call and Event-Driven OrchestrationCreating and Modifying an Orchestration WorkflowMonitoring the Orchestration WorkflowDistributed TransactionsChoreographed Transactions: The Saga PatternOrchestrated TransactionsCompensation WorkflowsSummary
Designing Function-Based Solutions as MicroservicesEnsure Strict Membership to a Bounded ContextCommit Offsets Only After Processing Has CompletedLess Is MoreChoosing a FaaS ProviderBuilding Microservices Out of FunctionsCold Start and Warm StartsStarting Functions with TriggersTriggering Based on New Events: The Event-Stream ListenerTriggering Based on Consumer Group LagTriggering on a ScheduleTriggering Using WebhooksTriggering on Resource EventsPerforming Business Work with FunctionsMaintaining StateFunctions Calling Other FunctionsEvent-Driven Communication PatternDirect-Call PatternTermination and ShutdownTuning Your FunctionsAllocating Sufficient ResourcesBatch Event-Processing ParametersScaling Your FaaS SolutionsSummary

Where Do BPCs Work Well?Integration with Existing and Legacy SystemsStateful Business Logic That Isn’t Reliant Upon Event OrderWhen the Data Layer Does Much of the WorkIndependent Scaling of the Processing and Data LayerHybrid BPC Applications with External Stream ProcessingExample: Using an External Stream-Processing Framework to Join Event StreamsSummary
A Brief History of Heavyweight FrameworksThe Inner Workings of Heavyweight FrameworksBenefits and LimitationsCluster Setup Options and Execution ModesUse a Hosted ServiceBuild Your Own Full ClusterCreate Clusters with CMS IntegrationApplication Submission ModesDriver ModeCluster ModeHandling State and Using CheckpointsScaling Applications and Handling Event Stream PartitionsScaling an Application While It Is RunningScaling an Application by Restarting ItAutoscaling ApplicationsRecovering from FailuresMultitenancy ConsiderationsLanguages and SyntaxChoosing a FrameworkExample: Session Windowing of Clicks and ViewsSummary
Benefits and LimitationsLightweight ProcessingHandling State and Using ChangelogsScaling Applications and Recovering from FailuresEvent ShufflingState AssignmentState Replication and Hot ReplicasChoosing a Lightweight FrameworkApache Kafka StreamsApache Samza: Embedded ModeLanguages and SyntaxStream-Table-Table Join: Enrichment PatternSummary
Handling External EventsAutonomously Generated EventsReactively Generated EventsHandling Autonomously Generated Analytical EventsIntegrating with Third-Party Request-Response APIsProcessing and Serving Stateful DataServing Real-Time Requests with Internal State StoresServing Real-Time Requests with External State StoresHandling Requests Within an Event-Driven WorkflowProcessing Events for User InterfacesMicro-Frontends in Request-Response ApplicationsThe Benefits of MicrofrontendsComposition-Based MicroservicesEasy Alignment to Business RequirementsDrawbacks of MicrofrontendsPotentially Inconsistent UI Elements and StylingVarying Microfrontend PerformanceExample: Experience Search and Review ApplicationSummary
Microservice-to-Team Assignment SystemEvent Stream Creation and ModificationEvent Stream Metadata TaggingQuotasSchema RegistrySchema Creation and Modification NotificationsOffset ManagementPermissions and Access Control Lists for Event StreamsState Management and Application ResetConsumer Offset Lag MonitoringStreamlined Microservice Creation ProcessContainer Management ControlsCluster Creation and ManagementProgrammatic Bringup of Event BrokersProgrammatic Bringup of Compute ResourcesCross-Cluster Event Data ReplicationProgrammatic Bringup of ToolingDependency Tracking and Topology VisualizationTopology ExampleSummary
General Testing PrinciplesUnit-Testing Topology FunctionsStateless FunctionsStateful FunctionsTesting the TopologyTesting Schema Evolution and CompatibilityIntegration Testing of Event-Driven MicroservicesLocal Integration TestingCreate a Temporary Environment Within the Runtime of Your Test CodeCreate a Temporary Environment External to Your Test CodeIntegrate Hosted Services Using Mocking and Simulator OptionsIntegrate Remote Services That Have No Local OptionsFull Remote Integration TestingProgrammatically Create a Temporary Integration Testing EnvironmentTesting Using a Shared EnvironmentTesting Using the Production EnvironmentChoosing Your Full-Remote Integration Testing StrategySummary
Principles of Microservice DeploymentArchitectural Components of Microservice DeploymentContinuous Integration, Delivery, and Deployment SystemsContainer Management Systems and Commodity HardwareThe Basic Full-Stop Deployment PatternThe Rolling Update PatternThe Breaking Schema Change PatternEventual Migration via Two Event StreamsSynchronized Migration to the New Event StreamThe Blue-Green Deployment PatternSummary
Communication LayersBusiness Domains and Bounded ContextsShareable Tools and InfrastructureSchematized EventsData Liberation and the Single Source of TruthMicroservicesMicroservice Implementation OptionsTestingDeployingFinal Words