Modern Data Engineering with Apache Spark: A Hands-On Guide for Building Mission-Critical Streaming Applications

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

S. HainesModern Data Engineering with Apache Sparkhttps://doi.org/10.1007/978-1-4842-7452-1_11

11. Apache Kafka and Spark Structured Streaming

Scott Haines¹  

(1)

San Jose, CA, USA

 

The last chapter was an introduction to using Apache Spark Structured Streaming. You learned how the popular Redis database can be used to create structured in-memory event streams and explored how to write stateful streaming applications.

This chapter expands on the skills acquired in the last chapter, which included an introduction to using the core Structured Streaming APIs—the DataStreamReader and the DataStreamWriter, how to utilize application checkpoints to create stateful ...