book

Stream Processing with Apache Spark

by Gerard Maas, Francois Garillot

June 2019

Beginner to intermediate

452 pages

10h 42m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Foreword
Preface
Who Should Read This Book?Installing SparkLearning ScalaThe Way AheadBibliographyConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgmentsFrom GerardFrom François
I. Fundamentals of Stream Processing with Apache Spark
1. Introducing Stream Processing
What Is Stream Processing?Batch Versus Stream ProcessingThe Notion of Time in Stream ProcessingThe Factor of UncertaintySome Examples of Stream ProcessingScaling Up Data ProcessingMapReduceThe Lesson Learned: Scalability and Fault ToleranceDistributed Stream ProcessingStateful Stream Processing in a Distributed SystemIntroducing Apache SparkThe First Wave: Functional APIsThe Second Wave: SQLA Unified EngineSpark ComponentsSpark StreamingStructured StreamingWhere Next?
2. Stream-Processing Model
Sources and SinksImmutable Streams Defined from One AnotherTransformations and AggregationsWindow AggregationsTumbling WindowsSliding WindowsStateless and Stateful ProcessingStateful StreamsAn Example: Local Stateful Computation in ScalaA Stateless Definition of the Fibonacci Sequence as a Stream TransformationStateless or Stateful StreamingThe Effect of TimeComputing on Timestamped EventsTimestamps as the Provider of the Notion of TimeEvent Time Versus Processing TimeComputing with a WatermarkSummary
3. Streaming Architectures
Components of a Data PlatformArchitectural ModelsThe Use of a Batch-Processing Component in a Streaming ApplicationReferential Streaming ArchitecturesThe Lambda ArchitectureThe Kappa ArchitectureStreaming Versus Batch AlgorithmsStreaming Algorithms Are Sometimes Completely Different in NatureStreaming Algorithms Can’t Be Guaranteed to Measure Well Against Batch AlgorithmsSummary
4. Apache Spark as a Stream-Processing Engine
The Tale of Two APIsSpark’s Memory UsageFailure RecoveryLazy EvaluationCache HintsUnderstanding LatencyThroughput-Oriented ProcessingSpark’s Polyglot APIFast Implementation of Data AnalysisTo Learn More About SparkSummary
5. Spark’s Distributed Processing Model
Running Apache Spark with a Cluster ManagerExamples of Cluster ManagersSpark’s Own Cluster ManagerUnderstanding Resilience and Fault Tolerance in a Distributed SystemFault RecoveryCluster Manager Support for Fault ToleranceData Delivery SemanticsMicrobatching and One-Element-at-a-TimeMicrobatching: An Application of Bulk-Synchronous ProcessingOne-Record-at-a-Time ProcessingMicrobatching Versus One-at-a-Time: The Trade-OffsBringing Microbatch and One-Record-at-a-Time Closer TogetherDynamic Batch IntervalStructured Streaming Processing ModelThe Disappearance of the Batch Interval
6. Spark’s Resilience Model
Resilient Distributed Datasets in SparkSpark ComponentsSpark’s Fault-Tolerance GuaranteesTask Failure RecoveryStage Failure RecoveryDriver Failure RecoverySummary
A. References for Part I

II. Structured Streaming
7. Introducing Structured Streaming
First Steps with Structured StreamingBatch AnalyticsStreaming AnalyticsConnecting to a StreamPreparing the Data in the StreamOperations on Streaming DatasetCreating a QueryStart the Stream ProcessingExploring the DataSummary
8. The Structured Streaming Programming Model
Initializing SparkSources: Acquiring Streaming DataAvailable SourcesTransforming Streaming DataStreaming API Restrictions on the DataFrame APISinks: Output the Resulting DataformatoutputModequeryNameoptionoptionstriggerstart()Summary
9. Structured Streaming in Action
Consuming a Streaming SourceApplication LogicWriting to a Streaming SinkSummary
10. Structured Streaming Sources
Understanding SourcesReliable Sources Must Be ReplayableSources Must Provide a SchemaAvailable SourcesThe File SourceSpecifying a File FormatCommon OptionsCommon Text Parsing Options (CSV, JSON)JSON File Source FormatCSV File Source FormatParquet File Source FormatText File Source FormatThe Kafka SourceSetting Up a Kafka SourceSelecting a Topic Subscription MethodConfiguring Kafka Source OptionsKafka Consumer OptionsThe Socket SourceConfigurationOperationsThe Rate SourceOptions
11. Structured Streaming Sinks
Understanding SinksAvailable SinksReliable SinksSinks for ExperimentationThe Sink APIExploring Sinks in DetailThe File SinkUsing Triggers with the File SinkCommon Configuration Options Across All Supported File FormatsCommon Time and Date Formatting (CSV, JSON)The CSV Format of the File SinkThe JSON File Sink FormatThe Parquet File Sink FormatThe Text File Sink FormatThe Kafka SinkUnderstanding the Kafka Publish ModelUsing the Kafka SinkThe Memory SinkOutput ModesThe Console SinkOptionsOutput ModesThe Foreach SinkThe ForeachWriter InterfaceTCP Writer Sink: A Practical ForeachWriter ExampleThe Moral of this ExampleTroubleshooting ForeachWriter Serialization Issues
12. Event Time–Based Stream Processing
Understanding Event Time in Structured StreamingUsing Event TimeProcessing TimeWatermarksTime-Based Window AggregationsDefining Time-Based WindowsUnderstanding How Intervals Are ComputedUsing Composite Aggregation KeysTumbling and Sliding WindowsRecord DeduplicationSummary
13. Advanced Stateful Operations
Example: Car Fleet ManagementUnderstanding Group with State OperationsInternal State FlowUsing MapGroupsWithStateUsing FlatMapGroupsWithStateOutput ModesManaging State Over TimeSummary
14. Monitoring Structured Streaming Applications
The Spark Metrics SubsystemStructured Streaming MetricsThe StreamingQuery InstanceGetting Metrics with StreamingQueryProgressThe StreamingQueryListener InterfaceImplementing a StreamingQueryListener
15. Experimental Areas: Continuous Processing and Machine Learning
Continuous ProcessingUnderstanding Continuous ProcessingUsing Continuous ProcessingLimitationsMachine LearningLearning Versus ExploitingApplying a Machine Learning Model to a StreamExample: Estimating Room Occupancy by Using Ambient SensorsOnline Training
B. References for Part II
III. Spark Streaming
16. Introducing Spark Streaming
The DStream AbstractionDStreams as a Programming ModelDStreams as an Execution ModelThe Structure of a Spark Streaming ApplicationCreating the Spark Streaming ContextDefining a DStreamDefining Output OperationsStarting the Spark Streaming ContextStopping the Streaming ProcessSummary
17. The Spark Streaming Programming Model
RDDs as the Underlying Abstraction for DStreamsUnderstanding DStream TransformationsElement-Centric DStream TransformationsRDD-Centric DStream TransformationsCountingStructure-Changing TransformationsSummary
18. The Spark Streaming Execution Model
The Bulk-Synchronous ArchitectureThe Receiver ModelThe Receiver APIHow Receivers WorkThe Receiver’s Data FlowThe Internal Data ResilienceReceiver ParallelismBalancing Resources: Receivers Versus Processing CoresAchieving Zero Data Loss with the Write-Ahead LogThe Receiverless or Direct ModelSummary
19. Spark Streaming Sources
Types of SourcesBasic SourcesReceiver-Based SourcesDirect SourcesCommonly Used SourcesThe File SourceHow It WorksThe Queue SourceHow It WorksUsing a Queue Source for Unit TestingA Simpler Alternative to the Queue Source: The ConstantInputDStreamThe Socket SourceHow It WorksThe Kafka SourceUsing the Kafka SourceHow It WorksWhere to Find More Sources
20. Spark Streaming Sinks
Output OperationsBuilt-In Output OperationsprintsaveAsxyzforeachRDDUsing foreachRDD as a Programmable SinkThird-Party Output Operations
21. Time-Based Stream Processing
Window AggregationsTumbling WindowsWindow Length Versus Batch IntervalSliding WindowsSliding Windows Versus Batch IntervalSliding Windows Versus Tumbling WindowsUsing Windows Versus Longer Batch IntervalsWindow ReductionsreduceByWindowreduceByKeyAndWindowcountByWindowcountByValueAndWindowInvertible Window AggregationsSlicing StreamsSummary
22. Arbitrary Stateful Streaming Computation
Statefulness at the Scale of a StreamupdateStateByKeyLimitation of updateStateByKeyPerformanceMemory UsageIntroducing Stateful Computation with mapwithStateUsing mapWithStateEvent-Time Stream Computation Using mapWithState
23. Working with Spark SQL
Spark SQLAccessing Spark SQL Functions from Spark StreamingExample: Writing Streaming Data to ParquetDealing with Data at RestUsing Join to Enrich the Input StreamJoin OptimizationsUpdating Reference Datasets in a Streaming ApplicationEnhancing Our Example with a Reference DatasetSummary
24. Checkpointing
Understanding the Use of CheckpointsCheckpointing DStreamsRecovery from a CheckpointLimitationsThe Cost of CheckpointingCheckpoint Tuning
25. Monitoring Spark Streaming
The Streaming UIUnderstanding Job Performance Using the Streaming UIInput Rate ChartScheduling Delay ChartProcessing Time ChartTotal Delay ChartBatch DetailsThe Monitoring REST APIUsing the Monitoring REST APIInformation Exposed by the Monitoring REST APIThe Metrics SubsystemThe Internal Event BusInteracting with the Event BusSummary
26. Performance Tuning
The Performance Balance of Spark StreamingThe Relationship Between Batch Interval and Processing DelayThe Last Moments of a Failing JobGoing Deeper: Scheduling Delay and Processing DelayCheckpoint Influence in Processing TimeExternal Factors that Influence the Job’s PerformanceHow to Improve Performance?Tweaking the Batch IntervalLimiting the Data Ingress with Fixed-Rate ThrottlingBackpressureDynamic ThrottlingTuning the Backpressure PIDCustom Rate EstimatorA Note on Alternative Dynamic Handling StrategiesCachingSpeculative Execution
C. References for Part III
IV. Advanced Spark Streaming Techniques
27. Streaming Approximation and Sampling Algorithms
Exactness, Real Time, and Big DataExactnessReal-Time ProcessingBig DataThe Exactness, Real-Time, and Big Data triangleBig Data and Real TimeApproximation AlgorithmsHashing and Sketching: An IntroductionCounting Distinct Elements: HyperLogLogRole-Playing Exercise: If We Were a System AdministratorPractical HyperLogLog in SparkCounting Element Frequency: Count Min SketchesIntroducing Bloom FiltersBloom Filters with SparkComputing Frequencies with a Count-Min SketchRanks and Quantiles: T-DigestT-Digest in SparkReducing the Number of Elements: SamplingRandom SamplingStratified Sampling
28. Real-Time Machine Learning
Streaming Classification with Naive BayesstreamDM IntroductionNaive Bayes in PracticeTraining a Movie Review ClassifierIntroducing Decision TreesHoeffding TreesHoeffding Trees in Spark, in PracticeStreaming Clustering with Online K-MeansK-Means ClusteringOnline Data and K-MeansThe Problem of Decaying ClustersStreaming K-Means with Spark Streaming
D. References for Part IV
V. Beyond Apache Spark
29. Other Distributed Real-Time Stream Processing Systems
Apache StormProcessing ModelThe Storm TopologyThe Storm ClusterCompared to SparkApache FlinkA Streaming-First FrameworkCompared to SparkKafka StreamsKafka Streams Programming ModelCompared to SparkIn the CloudAmazon Kinesis on AWSMicrosoft Azure Stream AnalyticsApache Beam/Google Cloud Dataflow
30. Looking Ahead
Stay Plugged InSeek Help on Stack OverflowStart Discussions on the Mailing ListsAttend ConferencesAttend MeetupsRead BooksContributing to the Apache Spark Project
E. References for Part V
Index

Content preview from Stream Processing with Apache Spark

Part II. Structured Streaming

In this part, we examine Structured Streaming.

We begin our journey by exploring a practical example that should help you build your intuition for the model. From there, we examine the API and get into the details of the following aspects of stream processing:

Consuming data using sources
Building data-processing logic using the rich Streaming Dataframe/Dataset API
Understanding and working with event time
Dealing with state in streaming applications
Learning about arbitrary stateful transformations
Writing the results to other systems using sinks

Before closing, we provide an overview of the operational aspects of Structured Streaming.

Finally, we explore the current developments in this exciting new streaming API and provide insights into experimental areas like machine learning applications and near-real-time data processing with continuous streaming.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781491944233Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Stream Processing with Apache Spark

by Gerard Maas, Francois Garillot

Part II. Structured Streaming

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.