Modern Data Engineering with Apache Spark: A Hands-On Guide for Building Mission-Critical Streaming Applications

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

S. HainesModern Data Engineering with Apache Sparkhttps://doi.org/10.1007/978-1-4842-7452-1_13

13. Advanced Analytics with Spark Stateful Structured Streaming

Scott Haines¹  

(1)

San Jose, CA, USA

 

In the last chapter, you learned to use Apache Spark’s powerful aggregation and analytics functions, from the agg operator that enabled powerful columnar aggregation capabilities directly off a grouped dataset, to the analytical window functions that allowed you to partition and analyze datasets using these unique windowing capabilities. This gave you the ability to look back (lag) or forward (lead) across many rows from your current position in an active iteration. ...