Chapter 5: Introducing Delta Engine
Delta Engine is the query engine of Delta Lake, which is included by default in Azure Databricks. It is built in a way that allows us to optimize the processing of data in our Delta Lake in a variety of ways, thanks to optimized layouts and improved data indexing. These optimization operations include the use of dynamic file pruning (DFP), Z-Ordering, Auto Compaction, ad hoc processing, and more. The added benefit of these optimization operations is that several of these operations take place in an automatic manner, just by using Delta Lake. You will be using Delta Engine optimization in many ways.
In this chapter, you will learn how to make use of Delta Lake to optimize your Delta Lake ETL in Azure Databricks. ...
Get Distributed Data Systems with Azure Databricks now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
