book

Practical Lakehouse Architecture

by Gaurav Ashok Thalpati

July 2024

Intermediate to advanced

286 pages

7h 53m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Includes

Includes Quizzes

Who Should Read This Book?Why I Wrote This BookNavigating This BookO’Reilly Online LearningConventions Used in This BookHow to Contact UsAcknowledgments
Understanding Data ArchitectureWhat Is Data Architecture?How Does Data Architecture Help Build a Data Platform?Core Components of a Data PlatformWhy Do We Need a New Data Architecture?Lakehouse Architecture: A New PatternThe Lakehouse: Best of Both WorldsUnderstanding Lakehouse ArchitectureLakehouse Architecture CharacteristicsLakehouse Architecture BenefitsKey TakeawaysReferences
Traditional Architectures: Data Lakes and Data WarehousesData Warehouse FundamentalsData Lake FundamentalsModern Data PlatformsFinding Answers in the CloudStandalone ApproachCombined ApproachExpectations of Modern Data PlatformsComparison: Data Warehouse, Data Lake, LakehouseCapabilities and LimitationsImplementation ActivitiesAdministration and ManagementBusiness OutcomesLakehouse Architecture: The Default Choice for Future Data Platforms?Key TakeawaysReferences
Lakehouse Storage: Key ConceptsRow Versus Columnar StorageStorage-based Performance OptimizationLakehouse Storage ComponentsCloud Object StorageFile FormatsTable FormatsKey Design ConsiderationsEcosystem SupportCommunity SupportSupported File FormatsSupported Compute EnginesSupported FeaturesCommercial Product SupportCurrent and Future VersionsPerformance BenchmarkingComparisonsSharing FeaturesKey TakeawaysReferences
Understanding MetadataTechnical MetadataBusiness MetadataHow Metastores and Data Catalogs Work TogetherFeatures of a Data CatalogSearch, Explore, and Discover DataData ClassificationData Governance and SecurityData LineageUnified Data CatalogChallenges of Siloed Metadata ManagementWhat Is a Unified Data Catalog?Benefits of a Unified Data CatalogImplementing a Data Catalog: Key Design Considerations and OptionsUsing Hive metastoreUsing AWS ServicesUsing Azure ServicesUsing GCP ServicesUsing DatabricksKey TakeawaysReferences
Data Computation Benefits of Lakehouse ArchitectureIndependent ScalingCross-region, Cross-account AccessUnified Batch and Real-Time ProcessingEnhanced BI PerformanceFreedom to Choose Different Engine TypesCross-zone AnalysisCompute Engine Options for Lakehouse PlatformsOpen Source ToolsCloud ServicesThird-Party PlatformsKey Design ConsiderationsOpen Table Format SupportSupported Version and FeaturesEcosystem SupportPersona-Based PreferencesManaged Open Source Versus Cloud Native Versus Third-Party ProductsData Consumption WorkloadsKey TakeawaysReferences
What Is Data Governance and Data Security?Benefits of Data Governance and Data SecurityUnified Governance and Security in Lakehouse ArchitectureGovernance and Security Processes in Lakehouse ArchitectureMetadata ManagementCompliance and RegulationsData and ML Model QualityLineage Across Data and AI assetsData and AI Asset SharingData OwnershipAuditing and MonitoringAccess ManagementData ProtectionHandling Sensitive DataWhat’s Your Role?Key TakeawaysReferences
Pre-design ActivitiesUnderstanding Platform RequirementsStudying Existing SystemUnderstanding the Organization’s Vision and Data StrategyConducting Workshops and InterviewsChoosing the Right ArchitectureEstablishing Guiding PrinciplesData EcosystemScalability and PerformanceCost Control and OptimizationPlatform OperationsGovernance and SecurityDesign Considerations and Implementation Best PracticesArchitecture BlueprintData IngestionData StorageData ProcessingData Consumption and DeliveryCommon ServicesDesign ReferencesStep-by-Step Design GuideDesign QuestionnaireKey TakeawaysReferences
Delivering a Real-World LakehouseEstimation and Planning PhaseEstimationPlanningAnalysis and Design PhaseAnalyzing the Existing SystemData ModelingFinalizing the Tech StackImplementation and Test PhaseHistorical Data MigrationData Reconciliation and TestingReverse EngineeringData Quality and Handling Sensitive DataSupport and Maintenance PhaseAuditing and TrackingDisaster Recovery StrategyDecommissioning the Old SystemDelivery ReferencesProject DeliverablesReference ArchitecturesKey TakeawaysReferences
Warehouse to Lakehouse: What’s Next?Data MeshHTAPZero ETLInteroperability and New FormatsUniversal Format (UniForm)Apache XTableUpcoming File and Table FormatsManaged Platforms for Public and Private CloudsMicrosoft Fabric and Other PlatformsManaged Lakehouse for Private Cloud PlatformAI in a LakehouseKey TakeawaysBook ConclusionReferences

Overview

This concise yet comprehensive guide explains how to adopt a data lakehouse architecture to implement modern data platforms. It reviews the design considerations, challenges, and best practices for implementing a lakehouse and provides key insights into the ways that using a lakehouse can impact your data platform, from managing structured and unstructured data and supporting BI and AI/ML use cases to enabling more rigorous data governance and security measures.

Practical Lakehouse Architecture shows you how to:

Understand key lakehouse concepts and features like transaction support, time travel, and schema evolution
Understand the differences between traditional and lakehouse data architectures
Differentiate between various file formats and table formats
Design lakehouse architecture layers for storage, compute, metadata management, and data consumption
Implement data governance and data security within the platform
Evaluate technologies and decide on the best technology stack to implement the lakehouse for your use case
Make critical design decisions and address practical challenges to build a future-ready data platform
Start your lakehouse implementation journey and migrate data from existing systems to the lakehouse