Spark SQL
Spark SQL is a Spark module for structured data processing that was introduced in Spark 1.0. This module is a tightly integrated relational engine that inert-operates with the core Spark API. It enables data engineers to write applications that load structured data from disparate sources and join them as a unified, and possibly continuous, Excel-like data frames; and then they can implement complex ETL workflows and advanced analytics.
The Spark 2.0 release brought in significant unification of APIs and expanded the SQL capabilities, including support for subqueries. The Dataset API and DataFrames API are now unified, with DataFrames being a "kind" of Datasets. The unified APIs build the foundation for Spark's future, spanning across ...
Get Spark for Data Science now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
