The introduction of the delta file format within Azure Data Lake Storage gen2 has been a modern approach to managing changing records and data since regular parquet file formats are immutable, and there is no graceful method of performing CRUD operations on these native parquet file formats. Despite the advantages of delta format files in the Data Lake, this Change Data Capture process also comes with significant overhead of having to scan and read the entire files even if only a few records within have changed. Change ...
© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022
R. L'EsteveThe Azure Data Lakehouse Toolkithttps://doi.org/10.1007/978-1-4842-8233-5_88. Change Data Feed
Ron L’Esteve1
(1)
Chicago, IL, USA
Get The Azure Data Lakehouse Toolkit: Building and Scaling Data Lakehouses on Azure with Delta Lake, Apache Spark, Databricks, Synapse Analytics, and Snowflake now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
