book

Apache Polaris: The Definitive Guide

by Alex Merced, Andrew Madson, Tomer Shiran

September 2025

Beginner to intermediate

258 pages

5h 47m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Conventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
Modern Data ChallengesThe World of Data WarehousesMoving Forward with Data LakesThe Cloud RevolutionFile-Based Analytics with Apache ParquetThe Data Lakehouse SolutionThe Key Benefits of a Data LakehouseThe Path Forward: Data Lakehouse Table FormatsThe Role of Table FormatsThe Benefits of Table FormatsExisting Table FormatsApache IcebergWhat Is Apache Iceberg?Metadata File (metadata.json)Manifest ListManifest FilesData FilesDelete FilesConclusion
What Is and Isn’t an Apache Iceberg CatalogThe Mechanics of Apache Iceberg CatalogsTypes of Apache Iceberg CatalogsFile-System CatalogsService CatalogsChallenges of Diverse Catalog OptionsClient-Side ComplexityConfiguration ChallengesAuthorization ChallengesThe Need for a Unified ApproachThe Apache Iceberg REST Catalog SpecificationKey Benefits of the REST Catalog SpecificationThe Evolution of REST Catalog ImplementationsApache PolarisThe Birth of Apache PolarisPolaris: A New Era of Lakehouse CatalogsConclusion
What Is Polaris?CatalogsKey Features of Polaris CatalogsBenefits of Multi-Catalog ArchitecturePrincipalsWhat Are Principals?Managing PrincipalsPrincipal LifecycleCatalog RolesDefining Permissions in Catalog RolesAssigning Catalog Roles to PrincipalsBest Practices for Catalog RolesPrincipal RolesWhat Are Principal Roles?Benefits of Principal RolesBest Practices for Principal RolesPolaris Security Best PracticesMulti-Tenant EnvironmentsCross-Team CollaborationCompliance and Sensitive Data GovernanceCloud-Native DeploymentsConclusion
NessieWhat Makes Nessie Unique?Why Use Nessie with Polaris?Example: Nessie and Polaris in ActionGravitinoWhat Makes Gravitino Unique?Why Use Gravitino with Polaris?Example: Distributed Metadata GovernanceLakekeeperWhat Makes Lakekeeper Unique?Why Use Lakekeeper with Polaris?Example: Multi-Tenant Metadata GovernanceAWS GlueWhy Use the AWS Glue Catalog?Why Use Glue with Polaris?Example: Hybrid Team CollaborationConclusion
Catalog OperationsList CatalogsCreate a CatalogGet Catalog DetailsUpdate a CatalogDelete a CatalogPrincipal OperationsList PrincipalsCreate a PrincipalGet Principal DetailsUpdate a PrincipalDelete a PrincipalRotate Principal CredentialsManaging RolesCreate a Catalog RoleCreate a Principal RoleList Catalog RolesList Roles Assigned to a PrincipalList All Principal RolesList Principals Assigned to a Principal RoleGet Catalog Roles Mapped to a Principal RoleGet Details of a Principal RoleAdd a Grant to a Catalog RoleRevoke a Grant from a Catalog RoleAssign a Catalog Role to a Principal RoleAssign a Role to a PrincipalUpdate a Principal RoleRevoke a Role from a PrincipalRevoke a Catalog Role from a Principal RoleDelete a Principal RoleDelete a Catalog RoleApache Iceberg REST Catalog EndpointsConfiguration APIOAuth2 APITable APIView APIConclusion

Deploying Locally with DockerPrerequisitesStep 1: Clone the RepositoryStep 2: Configure Environment VariablesStep 3: Understand the Docker Compose FileStep 4: Starting the EnvironmentStep 5: Stopping the EnvironmentCreating CatalogsWhen to Create a CatalogCreating Catalog RolesWhen to Create Catalog RolesCreating PrincipalsCreating Principal RolesWhen to Create a Principal RoleAssigning the Catalog Role to the Principal Role and Setting Permissions on the CatalogSummary
Connecting Your Apache Polaris Catalog to Apache SparkUsing Spark Dataframe API with Apache Polaris (Incubating)Creating a TableQuerying a TableUpdating a TableDeleting RowsAppending DataReading Metadata TablesUsing SparkSQL with Apache PolarisCreating a TableQuerying a TableInserting DataUpdating DataDeleting DataMerging DataReading Metadata TablesTime Travel QueriesUsing Spark Streaming with Apache PolarisSetting Up Spark Streaming with PolarisStreaming Reads from PolarisStreaming Writes to PolarisHandling Deletes and OverwritesUsing Partitioned TablesMaintaining Streaming TablesConclusion
Establishing Connectivity Between Snowflake and PolarisConfiguring an External VolumeCreating a Polaris Catalog IntegrationQuerying Iceberg Tables via Snowflake and PolarisRegistering an Existing Polaris Table in SnowflakeQuerying the External Iceberg TableUsing Snowflake Open Catalog (Managed Polaris)Polaris-Backed Tables vs. Native Snowflake TablesConclusion
Connecting Dremio to an Apache Polaris CatalogConnecting Polaris Using the REST Catalog ConnectorConnecting Snowflake’s Open Catalog to DremioWhy Disable Use Vended Credentials?Using Dremio SQL with Apache PolarisQuerying Iceberg Tables via PolarisQuerying the Iceberg Metadata TablesCreating Tables and CTAS in Polaris via DremioAdding Data from Files to a Table Using Copy IntoMaintaining Your Iceberg Tables with DremioDremio Automates OptimizationConclusion
Using the Polaris CLICLI Structure, Authentication, and ProfilesManaging Entities with the CLIUnderstanding RealmsObservability: Metrics, Tracing, and LoggingMetrics with Micrometer and PrometheusTracing with OpenTelemetryLogging and Debugging with QuarkusConfiguring Polaris for ProductionSecurity and Authentication ConfigurationDurable Metadata with MetastoresHardening Defaults and Managing Feature FlagsScaling, Concurrency, and Rate LimitsFinalizing and Verifying Your Production SetupConclusion
Managed PolarisThe REST Catalog EcosystemData Processing EnginesStreaming and Ingestion PlatformsOther Data-Stack ToolsThe Apache Polaris RoadmapGeneric Table SupportPolicy StoreTable Maintenance FrameworkSQL and NoSQL PersistenceS3-Compatible Storage SupportCatalog UIFederated CatalogsFederated Role SupportPolaris Event ListenersUnstructured Data in PolarisConclusion

Content preview from Apache Polaris: The Definitive Guide

Chapter 8. Using Apache Polaris with Snowflake

In this chapter, we explore how to integrate Apache Polaris with Snowflake to query Iceberg tables from Snowflake’s platform. We’ll walk through setting up Snowflake to connect to a Polaris catalog (either a self-hosted Polaris OSS instance or Snowflake’s managed Polaris service, Open Catalog), configuring the necessary external resources, running SQL queries on Polaris-managed Iceberg tables from Snowflake, and understanding the differences between Polaris-backed tables and native Snowflake tables. By the end, you will be able to query Iceberg tables via Snowflake using Polaris as the metadata catalog. You can also appreciate the trade-offs between this approach and Snowflake’s native table storage.

You should have a running Apache Polaris service with an Iceberg catalog and at least one table available. This can be a self-hosted Polaris instance (as set up in previous chapters) or the Snowflake Open Catalog. You will also need a Snowflake account with appropriate permissions (ACCOUNTADMIN or ORGADMIN role to create integrations) and access to the cloud storage where the Iceberg data resides.

Establishing Connectivity Between Snowflake and Polaris

To allow Snowflake to query data managed by Apache Polaris, it needs to connect to Polaris’s REST Catalog API. Snowflake treats Polaris as an external Iceberg catalog, retrieving table metadata and reading data files from cloud storage. This integration is achieved via two Snowflake objects: ...