Modern Data Engineering with Apache Spark: A Hands-On Guide for Building Mission-Critical Streaming Applications

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

S. HainesModern Data Engineering with Apache Sparkhttps://doi.org/10.1007/978-1-4842-7452-1_10

10. Patterns for Writing Structured Streaming Applications

Scott Haines¹  

(1)

San Jose, CA, USA

 

In the last chapter, we built up a streaming frame of reference for working with data in-flight. By introducing common problems and pitfalls that tend to occlude the path to streaming success, you discovered the nuances between local and global coordinated time, aka the trouble with time. You also learned about the complex problems that arise when corrupt or invalid data makes its way into your data network, in data problems in flight. In a perfect world, each event or ...