Apache Iceberg: The Definitive Guide

Chapter 10. Apache Iceberg in Production

Data engineers are responsible for collecting, storing, and processing data in a way that is efficient, reliable, and secure. When putting data into production, they need to follow a set of best practices to ensure that the data is accurate, consistent, and accessible. In this chapter, we will discuss many of the tools that can be used to help monitor and maintain Apache Iceberg tables in production. We will start with a discussion of Apache Iceberg metadata tables, which you can use to better understand your Iceberg tables. Then we will cover ways to ensure data quality, including branching to isolate ingestion at the table or catalog level; catalog versioning to carry out multitable transactions; and rolling back the state of a table or catalog when things go wrong.

All the practices discussed in this chapter can be applied in reactive or proactive ways. A reactive approach means reacting to situations that already exist, such as rewriting a partition that has already become too large or rolling back a table that has already ingested bad data.

Proactive techniques attempt to prevent problems like these in the first place and include monitoring partition sizes before they affect query performance using metadata tables, and using branching to isolate ingestion so that no bad data makes it into production before quality checks have occurred.

You will want to apply both approaches so that you can prevent problems from occurring and address ...