book

Learning and Operating Presto

by Angelica Lo Duca, Tim Meehan, Vivek Bharathan, Ying Su

September 2023

Intermediate to advanced

191 pages

4h 32m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Why We Wrote This BookWho This Book Is ForConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgmentsAngelica Lo DucaTim MeehanVivek BharathanYing Su
Data Warehouses and Data LakesThe Role of Presto in a Data LakePresto Origins and Design ConsiderationsHigh PerformanceHigh ScalabilityCompliance with the ANSI SQL StandardFederation of Data SourcesRunning in the CloudPresto Architecture and Core ComponentsAlternatives to PrestoApache ImpalaApache HiveSpark SQLTrinoPresto Use CasesReporting and DashboardingAd Hoc QueryingETL Using SQLData LakehouseReal-Time Analytics with Real-Time DatabasesIntroducing Our Case StudyConclusion
Presto Manual InstallationRunning Presto on DockerInstalling DockerPresto Docker ImageBuilding and Running Presto on DockerThe Presto SandboxDeploying Presto on KubernetesIntroducing KubernetesConfiguring Presto on KubernetesAdding a New CatalogRunning the Deployment on KubernetesQuerying Your Presto InstanceListing CatalogsListing SchemasListing TablesQuerying a TableConclusion
Service Provider InterfaceConnector ArchitecturePopular ConnectorsThriftWriting a Custom ConnectorPrerequisitesPlugin and ModuleConfigurationMetadataInput/OutputDeploying Your ConnectorApache PinotSetting Up and Configuring PrestoPresto-Pinot Querying in ActionConclusion
Setting Up the EnvironmentPresto ClientDocker ImageKubernetes NodeConnectivity to PrestoREST APIPythonRJDBCNode.jsODBCOther Presto Client LibrariesBuilding a Client Dashboard in PythonSetting Up the ClientBuilding the DashboardConclusion
The Emergence of the LakehouseData Lakehouse ArchitectureData LakeFile StoreFile FormatTable FormatQuery EngineMetadata ManagementData GovernanceData Access ControlBuilding a Data LakehouseConfiguring MinIOConfiguring HMSConfiguring SparkRegistering Hudi Tables with HMSConnecting and Querying PrestoConclusion
Introducing Presto AdministrationConfigurationPropertiesSessionsJVMMonitoringConsoleREST APIMetricsManagementResource GroupsVerifiersSession Properties ManagersNamespace FunctionsConclusion
Introducing Presto SecurityBuilding Secure Communication in PrestoEncryptionKeystore ManagementConfiguring HTTPS/TLSAuthenticationFile-Based AuthenticationLDAPKerberosCreating a Custom AuthenticatorAuthorizationAuthorizing Access to the Presto REST APIConfiguring System Access ControlAuthorization Through Apache RangerConclusion
Introducing Performance TuningReasons for Performance TuningThe Performance Tuning Life CycleQuery Execution ModelApproaches for Performance Tuning in PrestoResource AllocationStorageQuery OptimizationAria ScanTable ScanningRepartitioningImplementing Performance TuningBuilding and Importing the Sample CSV Table in MinIOConverting the CSV Table in ORCDefining the Tuning ParametersRunning TestsConclusion
Introducing ScalabilityReasons to Scale PrestoCommon IssuesDesign ConsiderationsAvailabilityManageabilityPerformanceProtectionConfigurationHow to Scale PrestoMultiple CoordinatorsPresto on SparkSpillingUsing a Cloud ServiceConclusion

Content preview from Learning and Operating Presto

Chapter 5. Open Data Lakehouse Analytics

So far, you have learned how to connect Presto to a data lake using standard connectors such as MySQL and Pinot. In addition, you have learned how to write a custom connector using Presto’s Java classes and methods. Finally, you have connected a client to Presto to run generic or custom queries. Now it’s time to use Presto in an advanced, more realistic scenario that addresses the main challenges of big data management: table lookup, concurrent access to data, and access control.

In this chapter, we will give an overview of the data lakehouse and implement a practical scenario. The chapter is divided into two parts. In the first part, we introduce the architecture of a data lakehouse, focusing on its main components. In the second part of the chapter, you will implement a practical data lakehouse scenario using Presto and completely open components.

The Emergence of the Lakehouse

The first generation of data lakes, based primarily on the Hadoop Distributed File System (HDFS), demonstrated the promise of analytics at scale. As a result, many organizations formed data platform architectures consisting of data lakes and data warehouses, stitching pipelines and workflows between them. However, the resulting platform was very complex, with issues around reliability, data freshness, and cost.¹

To overcome these issues, organizations tried to stretch both the data lake and the data warehouse in terms of the workloads they could support, but with ...