High Performance Spark, 2nd Edition

Appendix A. Just Enough Iceberg and Friends

While there are many competing catalogs, there are some important common elements in terms of the representation of data on disk to support key operations. The ability to branch and time travel is an amazing tool for data quality work, but there are some different considerations with largely immutable (that is, nonchanging or nonchangeable) files. The different file layouts and approaches to query planning are important to understand—especially so that when you get poor performance, you can understand how to fix it. While these modern table formats offer more ACID-like behavior, it’s important to understand that they’re still not databases, and certain access patterns are generally slow.

Warning

Importantly, all of these table formats and catalogs are at relatively early stages of development, and different engines may support vastly different features, resulting in the same query producing different results between engines. This can go as far as one engine not supporting the format of deletes (or tombstones) from another engine and data that was intended to be deleted being returned in the query.

In this appendix, we will discuss the current setup of Iceberg and similar catalogs that we believe is most impactful for performance. However, since table formats are actively evolving at the time of writing, we’ve decided to make this largely exercise-focused. We encourage you to try these same exercises against your desired table format ...