© Ed Elliott 2021
E. ElliottIntroducing .NET for Apache Sparkhttps://doi.org/10.1007/978-1-4842-6992-3_9

9. Structured Streaming

Ed Elliott1  
(1)
Sussex, UK
 

In this chapter, we will look at an example of how to create a streaming application. Apache Spark’s structured streaming API allows you to use the DataFrame API to express your Apache Spark job. Instead of working with static datasets, you work with micro-batches of data using the scalable, fault-tolerant stream processing engine built on Apache Spark.

The application we will create will do two things. Firstly, it will examine every message for a specific condition and allow our application to raise an alert, and secondly it will gather all the data received within a 5-minute window, aggregate ...

Get Introducing .NET for Apache Spark: Distributed Processing for Massive Datasets now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.