While Spark offers tremendous value in the advanced analytics and big data spaces, there are currently a few known limitations around indexing with Spark when compared to the best-in-class SQL Server indexing systems and processes. While Spark isn’t great at b-tree indexing and single record lookups, Spark partitioning attempts to address some of these indexing limitations. However, when users query the data with a different search predicate than what was partitioned, this will result in a full scan of the data along ...
© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022
R. L'EsteveThe Azure Data Lakehouse Toolkithttps://doi.org/10.1007/978-1-4842-8233-5_1616. Hyperspace
Ron L’Esteve1
(1)
Chicago, IL, USA
Get The Azure Data Lakehouse Toolkit: Building and Scaling Data Lakehouses on Azure with Delta Lake, Apache Spark, Databricks, Synapse Analytics, and Snowflake now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
