O'Reilly logo

Learning PySpark by Denny Lee, Tomasz Drabas

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 10. Structured Streaming

This chapter will provide a jump-start on the concepts behind Spark Streaming and how this has evolved into Structured Streaming. An important aspect of Structured Streaming is that it utilizes Spark DataFrames. This shift in paradigm will make it easier for Python developers to start working with Spark Streaming.

In this chapter, your will learn:

  • What is Spark Streaming?
  • Why do we need Spark Streaming?
  • What is the Spark Streaming application data flow?
  • Simple streaming application using DStream
  • A quick primer on Spark Streaming global aggregations
  • Introducing Structured Streaming

Note, for the initial sections of this chapter, the example code used will be in Scala, as this was how most Spark Streaming code was written. ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required