book

Apache Polaris: The Definitive Guide

by Alex Merced, Andrew Madson, Tomer Shiran

September 2025

Beginner to intermediate

258 pages

5h 47m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Conventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
Modern Data ChallengesThe World of Data WarehousesMoving Forward with Data LakesThe Cloud RevolutionFile-Based Analytics with Apache ParquetThe Data Lakehouse SolutionThe Key Benefits of a Data LakehouseThe Path Forward: Data Lakehouse Table FormatsThe Role of Table FormatsThe Benefits of Table FormatsExisting Table FormatsApache IcebergWhat Is Apache Iceberg?Metadata File (metadata.json)Manifest ListManifest FilesData FilesDelete FilesConclusion
What Is and Isn’t an Apache Iceberg CatalogThe Mechanics of Apache Iceberg CatalogsTypes of Apache Iceberg CatalogsFile-System CatalogsService CatalogsChallenges of Diverse Catalog OptionsClient-Side ComplexityConfiguration ChallengesAuthorization ChallengesThe Need for a Unified ApproachThe Apache Iceberg REST Catalog SpecificationKey Benefits of the REST Catalog SpecificationThe Evolution of REST Catalog ImplementationsApache PolarisThe Birth of Apache PolarisPolaris: A New Era of Lakehouse CatalogsConclusion
What Is Polaris?CatalogsKey Features of Polaris CatalogsBenefits of Multi-Catalog ArchitecturePrincipalsWhat Are Principals?Managing PrincipalsPrincipal LifecycleCatalog RolesDefining Permissions in Catalog RolesAssigning Catalog Roles to PrincipalsBest Practices for Catalog RolesPrincipal RolesWhat Are Principal Roles?Benefits of Principal RolesBest Practices for Principal RolesPolaris Security Best PracticesMulti-Tenant EnvironmentsCross-Team CollaborationCompliance and Sensitive Data GovernanceCloud-Native DeploymentsConclusion
NessieWhat Makes Nessie Unique?Why Use Nessie with Polaris?Example: Nessie and Polaris in ActionGravitinoWhat Makes Gravitino Unique?Why Use Gravitino with Polaris?Example: Distributed Metadata GovernanceLakekeeperWhat Makes Lakekeeper Unique?Why Use Lakekeeper with Polaris?Example: Multi-Tenant Metadata GovernanceAWS GlueWhy Use the AWS Glue Catalog?Why Use Glue with Polaris?Example: Hybrid Team CollaborationConclusion
Catalog OperationsList CatalogsCreate a CatalogGet Catalog DetailsUpdate a CatalogDelete a CatalogPrincipal OperationsList PrincipalsCreate a PrincipalGet Principal DetailsUpdate a PrincipalDelete a PrincipalRotate Principal CredentialsManaging RolesCreate a Catalog RoleCreate a Principal RoleList Catalog RolesList Roles Assigned to a PrincipalList All Principal RolesList Principals Assigned to a Principal RoleGet Catalog Roles Mapped to a Principal RoleGet Details of a Principal RoleAdd a Grant to a Catalog RoleRevoke a Grant from a Catalog RoleAssign a Catalog Role to a Principal RoleAssign a Role to a PrincipalUpdate a Principal RoleRevoke a Role from a PrincipalRevoke a Catalog Role from a Principal RoleDelete a Principal RoleDelete a Catalog RoleApache Iceberg REST Catalog EndpointsConfiguration APIOAuth2 APITable APIView APIConclusion

Deploying Locally with DockerPrerequisitesStep 1: Clone the RepositoryStep 2: Configure Environment VariablesStep 3: Understand the Docker Compose FileStep 4: Starting the EnvironmentStep 5: Stopping the EnvironmentCreating CatalogsWhen to Create a CatalogCreating Catalog RolesWhen to Create Catalog RolesCreating PrincipalsCreating Principal RolesWhen to Create a Principal RoleAssigning the Catalog Role to the Principal Role and Setting Permissions on the CatalogSummary
Connecting Your Apache Polaris Catalog to Apache SparkUsing Spark Dataframe API with Apache Polaris (Incubating)Creating a TableQuerying a TableUpdating a TableDeleting RowsAppending DataReading Metadata TablesUsing SparkSQL with Apache PolarisCreating a TableQuerying a TableInserting DataUpdating DataDeleting DataMerging DataReading Metadata TablesTime Travel QueriesUsing Spark Streaming with Apache PolarisSetting Up Spark Streaming with PolarisStreaming Reads from PolarisStreaming Writes to PolarisHandling Deletes and OverwritesUsing Partitioned TablesMaintaining Streaming TablesConclusion
Establishing Connectivity Between Snowflake and PolarisConfiguring an External VolumeCreating a Polaris Catalog IntegrationQuerying Iceberg Tables via Snowflake and PolarisRegistering an Existing Polaris Table in SnowflakeQuerying the External Iceberg TableUsing Snowflake Open Catalog (Managed Polaris)Polaris-Backed Tables vs. Native Snowflake TablesConclusion
Connecting Dremio to an Apache Polaris CatalogConnecting Polaris Using the REST Catalog ConnectorConnecting Snowflake’s Open Catalog to DremioWhy Disable Use Vended Credentials?Using Dremio SQL with Apache PolarisQuerying Iceberg Tables via PolarisQuerying the Iceberg Metadata TablesCreating Tables and CTAS in Polaris via DremioAdding Data from Files to a Table Using Copy IntoMaintaining Your Iceberg Tables with DremioDremio Automates OptimizationConclusion
Using the Polaris CLICLI Structure, Authentication, and ProfilesManaging Entities with the CLIUnderstanding RealmsObservability: Metrics, Tracing, and LoggingMetrics with Micrometer and PrometheusTracing with OpenTelemetryLogging and Debugging with QuarkusConfiguring Polaris for ProductionSecurity and Authentication ConfigurationDurable Metadata with MetastoresHardening Defaults and Managing Feature FlagsScaling, Concurrency, and Rate LimitsFinalizing and Verifying Your Production SetupConclusion
Managed PolarisThe REST Catalog EcosystemData Processing EnginesStreaming and Ingestion PlatformsOther Data-Stack ToolsThe Apache Polaris RoadmapGeneric Table SupportPolicy StoreTable Maintenance FrameworkSQL and NoSQL PersistenceS3-Compatible Storage SupportCatalog UIFederated CatalogsFederated Role SupportPolaris Event ListenersUnstructured Data in PolarisConclusion

Content preview from Apache Polaris: The Definitive Guide

About the Authors

Alex Merced is head of DevRel at Dremio with experience as a developer and instructor. His professional journey includes roles at GenEd Systems, Crossfield Digital, CampusGuard, and General Assembly. He co-authored Apache Iceberg: The Definitive Guide published by O’Reilly and has spoken at notable events such as Data Day Texas and Data Council. Alex is passionate about technology, sharing his expertise through blogs, videos, podcasts like Datanation and Web Dev 101, and contributions to the JavaScript and Python communities with libraries like SencilloDB and CoquitoJS.

Andrew Madson is an experienced data leader with 17 years of experience leading technical teams. Currently the head of evangelism and education at Tobiko—the creators of SQLMesh and SQLGlot—Andrew has held senior leadership positions at institutions such as JP Morgan, LPL Financial, MassMutual, and Arizona State University. In addition to leading data teams, Andrew is a professor of data science and analytics at several universities, where he teaches graduate courses in machine learning, statistics, SQL, R, Python, Tableau, and Power BI.

Tomer Shiran is the founder and chief product officer of Dremio, an open data lakehouse platform that enables companies to run analytics in the cloud without the cost, complexity, and lock-in of data warehouses. As the company’s founding CEO, Tomer built a world-class organization that has raised over $400M and now serves hundreds of the world’s largest enterprises, ...