Delta Lake: The Definitive Guide

Chapter 5. Maintaining Your Delta Lake

The process of keeping our Delta Lake tables running efficiently over time is akin to any kind of preventative maintenance for a car or motorcycle or any alternative mode of transportation (a bike, a scooter, rollerblades). We wouldn’t wait for our tires to go flat before assessing the situation and finding a solution—we’d take action. We would start with simple observations, look for leaks, and ask ourselves, “Does the tire need to be patched? Could the problem be as simple as adding more air, or is this situation more dire, and the whole tire will need to be replaced?” The process of monitoring the situation, finding a remedy when we detect a problem, and applying the solution can be applied to our Delta Lake tables as well and is all part of the general process of maintaining the tables. In essence, we just need to think in terms of cleaning, monitoring, tuning, repairing, and replacing.

In the sections that follow, we’ll learn to take advantage of the Delta Lake utility methods and learn about their associated configurations (aka table properties). We’ll walk through some common methods for cleaning, tuning, repairing, and replacing our tables, in order to lend a helping hand while optimizing the performance and health of our tables, and ultimately build a firm understanding of the cause-and-effect relationships among the actions we take.

Using Delta Lake Table Properties

Delta Lake provides many utility functions to assist with the ...