Skip to Content
View all events

Getting Started with Apache Iceberg

Published by O'Reilly Media, Inc.

Beginner content levelBeginner

Transactional storage for large analytical datasets

Data engineers working with tools of the Hadoop ecosystem have been facing significant challenges when dealing with large datasets.

Querying and updating Hive tables may take a long time due to expensive I/O operations, while partitioning accommodates only certain query patterns and is not as effective as full indexing. Schema changes may require rewrites of entire tables, and due to the lack of ACID guarantees in Hive tables, modifying them in parallel by multiple independent processes can be difficult, if not impossible, to orchestrate.

Apache Iceberg is a new transactional table format used for handling huge analytical datasets and has become an industry standard for storing data in object stores and distributed file systems. In addition to ensuring the correctness of your data, Apache Iceberg allows data engineers to simplify existing architectures, make data processing jobs more efficient and unlock fundamentally new use cases for data lakes and data meshes.

Through concise presentations and interactive exercises, this course reviews the fundamentals of Apache Iceberg and the advantages it offers over plain Hive tables. The course teaches you how to create Iceberg tables and query them efficiently; how to migrate existing datasets to Iceberg; how to incorporate Iceberg into existing and new data pipelines; how to integrate Iceberg tables with popular data processing and querying engines; and how to use advanced features like schema and partition evolution. The course also shows you how to access table history to retrieve and recover data as it was at any point in time.

What you’ll learn and how you can apply it

By the end of this live online course, you’ll understand:

  • Features and fundamentals of Apache Iceberg
  • Common issues in building complex data processing workflows based on standard Hive tables
  • How to make improvements in development speed, execution times and reliability which ACID (Atomic, Consistent, Isolated, Durable) guarantees can bring to many data engineering efforts
  • How Iceberg brings these guarantees to distributed data processing architectures
  • How Iceberg tables can be integrated with popular data processing and querying engines
  • The pros and cons of using Iceberg and how to balance them against each other

And you’ll be able to:

  • Create Iceberg tables with optimal configuration and query them efficiently
  • Migrate existing datasets to Iceberg without copying the original data
  • Integrate Iceberg as a data source and data sink for Apache Spark, Apache Flink and Trino
  • Interact with Iceberg tables using Scala and Python
  • Take advantage of advanced capabilities like schema and partition evolution
  • Take advantage of Iceberg’s capability to “time travel” between historical snapshots of the data they store

This live event is for you because...

  • You are a data engineer looking to improve your understanding of useful database concepts
  • You would like to take advantage of ACID properties in data pipelines built using tools from the Hadoop ecosystem
  • You work with data scientists who will find dataset versioning of great help in making their experiments reproducible
  • You struggle with implementing performant, update-heavy operations like schema changes and partition evolution on large datasets
  • You would like to improve the performance of queries run using Apache Spark and query engines like Trino
  • You work with any of the following: Apache Iceberg, Apache Spark, Trino (aka PrestoSQL), Apache Parquet, Apache Avro, Apache Hive, Apache Hadoop

Prerequisites

  • Familiarity with any modern programming language, such as Java or Python.
  • Some prior experience building data pipelines, ideally using some of the components of the Hadoop ecosystem.
  • Prior experience with the Hadoop ecosystem, Spark, Flink or Trino will be useful, but is not required.

Recommended preparation

Recommended follow-up

Schedule

The time frames are only estimates and may vary according to how the class is progressing.

Introduction (15 minutes)

  • Presentation: Introduction and training overview
  • Exercise: Confirm technical requirements and preparation

Issues around ACID Properties and Hive Tables (25 minutes)

  • Presentation: Overview of ACID properties: definitions and examples
  • Overview of the plain Hive table layout
  • Practical examples of why the two do not mix well together
  • Discussion: Do you understand the presented concepts?
  • Are you familiar with the practical issues with the lack of ACID properties?
  • In what scenarios have you encountered them?
  • Q&A
  • Break

Introduction to Apache Iceberg (20 minutes)

  • Presentation: Short history of the Iceberg format: the thinking behind it and what needs it was supposed to fulfil
  • Main features of Iceberg: time travel, snapshot isolation, robust type system, flexible schema evolution, partition scheme evolution
  • Existing use cases in the industry: Netflix, Expedia, and others
  • Q&A

Working with Apache Iceberg I (30 minutes)

  • Presentation: Iceberg’s Java and Python APIs
  • Compatibility matrix for each
  • Discussion of when each should or should not be used
  • Segway to higher level frameworks
  • Exercises: Creating an Iceberg table using PySpark (Spark / Python)
  • Populating the table with example data (Spark / Python)
  • Reading and examining the same table using the Iceberg Python API
  • Q&A
  • Break

Working with Apache Iceberg II (50 minutes)

  • Presentation: Options for deployment (cloud, local file systems, HDFS)
  • Integration (Spark, Flink, Trino, arbitrary JVM and Python code), including compatibility matrix and configuration
  • Strategies for converting existing tables into Iceberg
  • Exercises: Integrating Iceberg with Spark (querying and other table operations)
  • Integrating Iceberg with Trino (querying and other table operations)
  • Q&A
  • Break

Under The Hood (25 minutes)

  • Presentation: The structure and inner workings of the table format
  • Exercise: Analyse the file structure of the tables created during previous exercises, understanding the AVRO metadata
  • Q&A

Summary and Final Remarks (15 minutes)

  • Presentation: Recap of what we learned
  • Pros and cons of using Iceberg
  • Suggestions for additional reading
  • Final Q&A

Your Instructor

Michal Gancarski

Michal Gancarski has been a professional software and data engineer for almost a decade, gaining valuable experience from working at both startups and larger companies. He’s currently a senior data engineer at GROPYUS, focused on the modeling of connected data using RDF knowledge graphs. Previously, Michal worked at Zalando, where he took part in designing and building some of the core components of the company’s data lake and its auxiliary services. He holds a degree in mathematics and economics from the University of London.

Skill covered

Apache Iceberg