Modern Data Engineering with Apache Spark: A Hands-On Guide for Building Mission-Critical Streaming Applications

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

S. HainesModern Data Engineering with Apache Sparkhttps://doi.org/10.1007/978-1-4842-7452-1_3

3. Working with Data

Scott Haines¹  

(1)

San Jose, CA, USA

 

The last chapter introduced you to the Spark architecture and programming model. We took a quick tour of the core Spark components and APIs and finished up with an exercise that introduced you to the spark-shell and the DataFrame API. You also saw your first glimpse of the Spark SQL API, which empowers you to express complex analytical queries quickly and easily in a structured way. It also that cleanly abstracts away the underlying complexities when composing difficult SQL expressions.

In this chapter, we continue ...