Modern Data Engineering with Apache Spark: A Hands-On Guide for Building Mission-Critical Streaming Applications

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

S. HainesModern Data Engineering with Apache Sparkhttps://doi.org/10.1007/978-1-4842-7452-1_6

6. Data Discovery and the Spark SQL Catalog

Scott Haines¹  

(1)

San Jose, CA, USA

 

Being able to connect and work with the data systems and services that are included in most companies’ modern tech stack is a critical skill for data engineers. Lucky for you, Spark provides the mechanisms to work with and transform data so you can take action and solve problems, instead of writing and maintaining yet another piece of custom infrastructure code. By relying on the core capabilities of Spark, you learned to harness the power of JDBC to interoperate with data stored in a ...