Optimizing Your Apache Iceberg Lakehouse

Chapter 1. Exploring the Apache Iceberg Architecture

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book. Please note that the GitHub repo is available at https://github.com/starburstdata/optimizing-iceberg-book/tree/main.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

It’s time to dig deeper into the architecture underlying Apache Iceberg. Like all data lakehouse table formats, Iceberg is built on the model of collocating many large files with the same file format and logical structure in a repository, accessed as if they were a traditional RDBMS table. Unlike RDBMS technologies, data lakehouses clearly separate storage from compute. A repository full of data files and scalable processing capacity is not enough for lakehouses; we need metadata to complete the picture.

That metadata is persisted as files on the data lake repository alongside the data files, which are used when querying an Iceberg table. This chapter will explore the fundamental architecture of how that metadata is represented, why it is valuable, how it needs to be regularly maintained, and how it enables interoperability among multiple compute engines.

Metadata and Iceberg

Rich metadata ...