book

Grokking Streaming Systems

by Ning Wang, Josh Fischer

March 2022

Intermediate to advanced

312 pages

6h 29m

English

Manning Publications

Read now

Unlock full access

prefaceacknowledgmentsabout this bookabout the authors
What is stream processing?Streaming system examplesStreaming systems and real timeHow a streaming system worksApplicationsBackend servicesInside a backend serviceBatch processing systemsInside a batch processing systemStream processing systemsInside a stream processing systemThe advantages of multi-stage architectureThe multi-stage architecture in batch and stream processing systemsCompare the systemsA model stream processing systemSummaryExercise
The chief needs a fancy tollboothIt started as HTTP requests, and it failedAJ and Miranda take time to reflectAJ ponders about streaming systemsComparing backend service and streamingHow a streaming system could fitQueues: A foundational conceptData transfer via queuesOur streaming framework (the start of it)The Streamwork framework overviewZooming in on the Streamwork engineCore streaming conceptsMore details of the conceptsThe streaming job execution flowYour first streaming jobExecuting the jobInspecting the job executionLook inside the engineKeep events movingThe life of a data elementReviewing streaming conceptsSummaryExercises
The sensor is emitting more eventsEven in streaming, real time is hardNew concepts: Parallelism is importantNew concepts: Data parallelismNew concepts: Data execution independenceNew concepts: Task parallelismData parallelism vs. task parallelismParallelism and concurrencyParallelizing the jobParallelizing componentsParallelizing sourcesViewing job outputParallelizing operatorsViewing job outputEvents and instancesEvent orderingEvent groupingShuffle groupingShuffle grouping: Under the hoodFields groupingFields grouping: Under the hoodEvent grouping executionLook inside the engine: Event dispatcherApplying fields grouping in your jobEvent orderingComparing grouping behaviorsSummaryExercises

A credit card fraud detection systemMore about the credit card fraud detection systemThe fraud detection businessStreaming isn’t always a straight lineZoom into the systemThe fraud detection job in detailNew conceptsUpstream and downstream componentsStream fan-out and fan-inGraph, directed graph, and DAGDAG in stream processing systemsAll new concepts in one pageStream fan-out to the analyzersLook inside the engineThere is a problem: EfficiencyStream fan-out with different streamsLook inside the engine againCommunication between the components via channelsMultiple channelsStream fan-in to the score aggregatorStream fan-in in the engineA brief introduction to anotherstream fan-in: JoinLook at the whole systemGraph and streaming jobsThe example systemsSummaryExercises
The latency requirement of the fraud detection systemRevisit the fraud detection jobAbout accuracyPartial resultA new streaming job to monitor system usageThe new system usage jobThe requirements of the new system usage jobNew concepts: (The number of) times delivered and times processedNew concept: Delivery semanticsChoosing the right semanticsAt-most-onceThe fraud detection jobAt-least-onceAt-least-once with acknowledgingTrack eventsHandle event processing failuresTrack early out eventsAcknowledging code in componentsNew concept: CheckpointingNew concept: StateCheckpointing in the system usage job for the at-least-once semanticCheckpointing and state manipulation functionsState handling code in the transaction source componentExactly-once or effectively-once?Bonus concept: Idempotent operationExactly-once, finallyState handling code in the system usage analyzer componentComparing the delivery semantics againSummaryExercisesUp next ...
Streaming system piecesParallelization and event groupingDAGs and streaming jobsDelivery semantics (guarantees)Delivery semantics used in the credit card fraud detection systemWhich way to go from hereWindowed computationsJoining data in real timeBackpressureStateless and stateful computations
Slicing up real-time dataBreaking down the problem in detailBreaking down the problem in detail (continued)Two different contextsWindowing in the fraud detection jobWhat exactly are windows?Looking closer into the windowNew concept: Windowing strategyFixed windowsFixed windows in the windowed proximity analyzerDetecting fraud with a fixed time windowFixed windows: Time vs. countSliding windowsSliding windows: Windowed proximity analyzerDetecting fraud with a sliding windowSession windowsSession windows (continued)Detecting fraud with session windowsSummary of windowing strategiesSlicing an event stream into data setsWindowing: Concept or implementationAnother lookKey–value store 101Implement the windowed proximity analyzerEvent time and other times for eventsWindowing watermarkLate eventsSummaryExercise
Joining emission data on the flyThe emissions job version 1The emission resolverAccuracy becomes an issueThe enhanced emissions jobFocusing on the joinWhat is a join again?How the stream join worksStream join is a different kind of fan-inVehicle events vs. temperature eventsTable: A materialized view of streamingVehicle events are less efficient to be materializedData integrity quickly became an issueWhat’s the problem with this join operator?Inner joinOuter joinThe inner join vs. outer joinDifferent types of joinsOuter joins in streaming systemsA new issue: Weak connectionWindowed joinsJoining two tables instead of joining a stream and tableRevisiting the materialized viewSummary
Reliability is criticalReview the systemStreamlining streaming jobsNew concepts: Capacity, utilization, and headroomMore about utilization and headroomNew concept: BackpressureMeasure capacity utilizationBackpressure in the Streamwork engineBackpressure in the Streamwork engine: PropagationOur streaming job during a backpressureBackpressure in distributed systemsNew concept: Backpressure watermarksAnother approach to handle lagging instances: Dropping eventsWhy do we want to drop events?Backpressure could be a symptom when the underlying issue is permanentStopping and resuming may lead to thrashing if the issue is permanentHandle thrashingSummary
The migration of the streaming jobsStateful components in the system usage jobRevisit: StateThe states in different componentsState data vs. temporary dataStateful vs. stateless components: The codeThe stateful source and operator in the system usage jobStates and checkpointsCheckpoint creation: Timing is hardEvent-based timingCreating checkpoints with checkpoint eventsA checkpoint event is handled by instance executorsA checkpoint event flowing through a jobCreating checkpoints with checkpoint events at the instance levelCheckpoint event synchronizationCheckpoint loading and backward compatibilityCheckpoint storageStateful vs. stateless componentsManually managed instance statesLambda architectureSummaryExercises
Is this really the end?Windowed computationsThe major window typesJoining data in real timeSQL vs. stream joinsInner joins vs. outer joinsUnexpected things can happen in streaming systemsBackpressure: Slow down sources or upstream componentsAnother approach to handle lagging instances: Dropping eventsBackpressure can be a symptom when the underlying issue is permanentStateful components with checkpointsEvent-based timingStateful vs. stateless componentsYou did it!

Content preview from Grokking Streaming Systems

5 Delivery semantics

In this chapter

introducing delivery semantics and their impact
at-most-once delivery semantic
at-least-once delivery semantic
exactly-once delivery semanticdelivery semantics.

“There’s never enough time to do it right, but there’s always enough time to do it over.”

—Jack Bergman

Computers are pretty good at performing accurate calculations. However, when computers work together in a distributed system, like many streaming systems, accuracy becomes a little bit more (I mean, a lot more) complicated. Sometimes, we may not want 100% accuracy because other more important requirements need to be met. “Why would we want wrong answers?” you might ask. This is a great question, and it is the one that we need to ask when designing ...