book

Practical Lakehouse Architecture

by Gaurav Ashok Thalpati

July 2024

Intermediate to advanced

286 pages

7h 53m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Includes

Includes Quizzes

Who Should Read This Book?Why I Wrote This BookNavigating This BookO’Reilly Online LearningConventions Used in This BookHow to Contact UsAcknowledgments
Understanding Data ArchitectureWhat Is Data Architecture?How Does Data Architecture Help Build a Data Platform?Core Components of a Data PlatformWhy Do We Need a New Data Architecture?Lakehouse Architecture: A New PatternThe Lakehouse: Best of Both WorldsUnderstanding Lakehouse ArchitectureLakehouse Architecture CharacteristicsLakehouse Architecture BenefitsKey TakeawaysReferences
Traditional Architectures: Data Lakes and Data WarehousesData Warehouse FundamentalsData Lake FundamentalsModern Data PlatformsFinding Answers in the CloudStandalone ApproachCombined ApproachExpectations of Modern Data PlatformsComparison: Data Warehouse, Data Lake, LakehouseCapabilities and LimitationsImplementation ActivitiesAdministration and ManagementBusiness OutcomesLakehouse Architecture: The Default Choice for Future Data Platforms?Key TakeawaysReferences
Lakehouse Storage: Key ConceptsRow Versus Columnar StorageStorage-based Performance OptimizationLakehouse Storage ComponentsCloud Object StorageFile FormatsTable FormatsKey Design ConsiderationsEcosystem SupportCommunity SupportSupported File FormatsSupported Compute EnginesSupported FeaturesCommercial Product SupportCurrent and Future VersionsPerformance BenchmarkingComparisonsSharing FeaturesKey TakeawaysReferences
Understanding MetadataTechnical MetadataBusiness MetadataHow Metastores and Data Catalogs Work TogetherFeatures of a Data CatalogSearch, Explore, and Discover DataData ClassificationData Governance and SecurityData LineageUnified Data CatalogChallenges of Siloed Metadata ManagementWhat Is a Unified Data Catalog?Benefits of a Unified Data CatalogImplementing a Data Catalog: Key Design Considerations and OptionsUsing Hive metastoreUsing AWS ServicesUsing Azure ServicesUsing GCP ServicesUsing DatabricksKey TakeawaysReferences
Data Computation Benefits of Lakehouse ArchitectureIndependent ScalingCross-region, Cross-account AccessUnified Batch and Real-Time ProcessingEnhanced BI PerformanceFreedom to Choose Different Engine TypesCross-zone AnalysisCompute Engine Options for Lakehouse PlatformsOpen Source ToolsCloud ServicesThird-Party PlatformsKey Design ConsiderationsOpen Table Format SupportSupported Version and FeaturesEcosystem SupportPersona-Based PreferencesManaged Open Source Versus Cloud Native Versus Third-Party ProductsData Consumption WorkloadsKey TakeawaysReferences
What Is Data Governance and Data Security?Benefits of Data Governance and Data SecurityUnified Governance and Security in Lakehouse ArchitectureGovernance and Security Processes in Lakehouse ArchitectureMetadata ManagementCompliance and RegulationsData and ML Model QualityLineage Across Data and AI assetsData and AI Asset SharingData OwnershipAuditing and MonitoringAccess ManagementData ProtectionHandling Sensitive DataWhat’s Your Role?Key TakeawaysReferences
Pre-design ActivitiesUnderstanding Platform RequirementsStudying Existing SystemUnderstanding the Organization’s Vision and Data StrategyConducting Workshops and InterviewsChoosing the Right ArchitectureEstablishing Guiding PrinciplesData EcosystemScalability and PerformanceCost Control and OptimizationPlatform OperationsGovernance and SecurityDesign Considerations and Implementation Best PracticesArchitecture BlueprintData IngestionData StorageData ProcessingData Consumption and DeliveryCommon ServicesDesign ReferencesStep-by-Step Design GuideDesign QuestionnaireKey TakeawaysReferences
Delivering a Real-World LakehouseEstimation and Planning PhaseEstimationPlanningAnalysis and Design PhaseAnalyzing the Existing SystemData ModelingFinalizing the Tech StackImplementation and Test PhaseHistorical Data MigrationData Reconciliation and TestingReverse EngineeringData Quality and Handling Sensitive DataSupport and Maintenance PhaseAuditing and TrackingDisaster Recovery StrategyDecommissioning the Old SystemDelivery ReferencesProject DeliverablesReference ArchitecturesKey TakeawaysReferences
Warehouse to Lakehouse: What’s Next?Data MeshHTAPZero ETLInteroperability and New FormatsUniversal Format (UniForm)Apache XTableUpcoming File and Table FormatsManaged Platforms for Public and Private CloudsMicrosoft Fabric and Other PlatformsManaged Lakehouse for Private Cloud PlatformAI in a LakehouseKey TakeawaysBook ConclusionReferences

Content preview from Practical Lakehouse Architecture

Chapter 8. Lakehouse in the Real World

We are approaching the end of this book and, so far, we’ve discussed various components, design considerations, tools and technologies, and best practices for implementing a lakehouse platform. In all these discussions, we’ve looked at the ideal design approaches and optimized solutions for building a lakehouse. However, as we all know, reality is much different than the ideal world. Implementing a lakehouse in the real world comes with several unforeseen challenges. To overcome these challenges, you will have to make some practical decisions and technical adjustments based on your judgment and experience.

In this chapter, I’ll discuss what it means to build a lakehouse in the real world—a world where projects get delayed by months (if not years), requirements keep changing, budgets keep reducing, and old platforms never get decommissioned! We will discuss several technical and non-technical challenges across the planning, design, implementation, and support phases of a lakehouse project.

Building a data platform is a long journey that can take several months to years. In the sections that follow, I’ll guide you through estimating, planning, and executing your lakehouse project and help you overcome some common, practical challenges. I’ll also provide you with a list of project deliverables and reference architectures that can help you kickstart your lakehouse journey.