Databricks Certified Data Engineer Associate Study Guide

Chapter 3. Transforming Data with Apache Spark

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 4th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

Databricks platform provides numerous transformative capabilities powered by Apache Spark. In this chapter, we will navigate through various data transformations tasks such as querying data files, writing to tables with various strategies, and performing advanced ETL operations. Moreover, we will discover the potential of higher-order functions, and SQL User Defined Functions (UDFs) in Spark.

Querying Data Files

Querying files in Databricks is a fundamental aspect of data exploration and analysis. In this section, we will explore the process of querying file content using SQL-like syntax. The primary mechanism for this is the SELECT statement, which allows us to query files directly to extract the files content.

To initiate a file query, we use the SELECT * FROM syntax, followed by the file format and the path to the file, as illustrated in Figure 3-1. It’s important to note that the ...