book

Practical Lakehouse Architecture

by Gaurav Ashok Thalpati

July 2024

Intermediate to advanced

286 pages

7h 53m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Includes

Includes Quizzes

Who Should Read This Book?Why I Wrote This BookNavigating This BookO’Reilly Online LearningConventions Used in This BookHow to Contact UsAcknowledgments
Understanding Data ArchitectureWhat Is Data Architecture?How Does Data Architecture Help Build a Data Platform?Core Components of a Data PlatformWhy Do We Need a New Data Architecture?Lakehouse Architecture: A New PatternThe Lakehouse: Best of Both WorldsUnderstanding Lakehouse ArchitectureLakehouse Architecture CharacteristicsLakehouse Architecture BenefitsKey TakeawaysReferences
Traditional Architectures: Data Lakes and Data WarehousesData Warehouse FundamentalsData Lake FundamentalsModern Data PlatformsFinding Answers in the CloudStandalone ApproachCombined ApproachExpectations of Modern Data PlatformsComparison: Data Warehouse, Data Lake, LakehouseCapabilities and LimitationsImplementation ActivitiesAdministration and ManagementBusiness OutcomesLakehouse Architecture: The Default Choice for Future Data Platforms?Key TakeawaysReferences
Lakehouse Storage: Key ConceptsRow Versus Columnar StorageStorage-based Performance OptimizationLakehouse Storage ComponentsCloud Object StorageFile FormatsTable FormatsKey Design ConsiderationsEcosystem SupportCommunity SupportSupported File FormatsSupported Compute EnginesSupported FeaturesCommercial Product SupportCurrent and Future VersionsPerformance BenchmarkingComparisonsSharing FeaturesKey TakeawaysReferences
Understanding MetadataTechnical MetadataBusiness MetadataHow Metastores and Data Catalogs Work TogetherFeatures of a Data CatalogSearch, Explore, and Discover DataData ClassificationData Governance and SecurityData LineageUnified Data CatalogChallenges of Siloed Metadata ManagementWhat Is a Unified Data Catalog?Benefits of a Unified Data CatalogImplementing a Data Catalog: Key Design Considerations and OptionsUsing Hive metastoreUsing AWS ServicesUsing Azure ServicesUsing GCP ServicesUsing DatabricksKey TakeawaysReferences
Data Computation Benefits of Lakehouse ArchitectureIndependent ScalingCross-region, Cross-account AccessUnified Batch and Real-Time ProcessingEnhanced BI PerformanceFreedom to Choose Different Engine TypesCross-zone AnalysisCompute Engine Options for Lakehouse PlatformsOpen Source ToolsCloud ServicesThird-Party PlatformsKey Design ConsiderationsOpen Table Format SupportSupported Version and FeaturesEcosystem SupportPersona-Based PreferencesManaged Open Source Versus Cloud Native Versus Third-Party ProductsData Consumption WorkloadsKey TakeawaysReferences
What Is Data Governance and Data Security?Benefits of Data Governance and Data SecurityUnified Governance and Security in Lakehouse ArchitectureGovernance and Security Processes in Lakehouse ArchitectureMetadata ManagementCompliance and RegulationsData and ML Model QualityLineage Across Data and AI assetsData and AI Asset SharingData OwnershipAuditing and MonitoringAccess ManagementData ProtectionHandling Sensitive DataWhat’s Your Role?Key TakeawaysReferences
Pre-design ActivitiesUnderstanding Platform RequirementsStudying Existing SystemUnderstanding the Organization’s Vision and Data StrategyConducting Workshops and InterviewsChoosing the Right ArchitectureEstablishing Guiding PrinciplesData EcosystemScalability and PerformanceCost Control and OptimizationPlatform OperationsGovernance and SecurityDesign Considerations and Implementation Best PracticesArchitecture BlueprintData IngestionData StorageData ProcessingData Consumption and DeliveryCommon ServicesDesign ReferencesStep-by-Step Design GuideDesign QuestionnaireKey TakeawaysReferences
Delivering a Real-World LakehouseEstimation and Planning PhaseEstimationPlanningAnalysis and Design PhaseAnalyzing the Existing SystemData ModelingFinalizing the Tech StackImplementation and Test PhaseHistorical Data MigrationData Reconciliation and TestingReverse EngineeringData Quality and Handling Sensitive DataSupport and Maintenance PhaseAuditing and TrackingDisaster Recovery StrategyDecommissioning the Old SystemDelivery ReferencesProject DeliverablesReference ArchitecturesKey TakeawaysReferences
Warehouse to Lakehouse: What’s Next?Data MeshHTAPZero ETLInteroperability and New FormatsUniversal Format (UniForm)Apache XTableUpcoming File and Table FormatsManaged Platforms for Public and Private CloudsMicrosoft Fabric and Other PlatformsManaged Lakehouse for Private Cloud PlatformAI in a LakehouseKey TakeawaysBook ConclusionReferences

Content preview from Practical Lakehouse Architecture

About the Author

Gaurav Thalpati is an independent consultant with over two decades of experience building data and analytics platforms. He has worked on various data projects and played different roles, including ETL/BI developer, data engineer, data analyst, and data architect. Based in Pune, India, Gaurav is passionate about sharing his knowledge with other data practitioners and guiding them in designing and implementing scalable and cost-effective data platforms.