The Cloud Data Lake

The Cloud Data Lake

by Rukmani Gopalan
Released December 2022
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781098116583

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Buy on Amazon Buy on ebooks.com
Start your free trial

Book description

More organizations than ever understand the importance of data lake architectures for deriving value from their data. Building a robust, scalable, and performant data lake remains a complex proposition, however, with a buffet of tools and options that need to work together to provide a seamless end-to-end pipeline from data to insights.

This book provides a concise yet comprehensive overview on the setup, management, and governance of a cloud data lake. Author Rukmani Gopalan, a product management leader and data enthusiast, guides data architects and engineers through the major aspects of working with a cloud data lake, from design considerations and best practices to data format optimizations, performance optimization, cost management, and governance.

  • Learn the benefits of a cloud-based big data strategy for your organization
  • Get guidance and best practices for designing performant and scalable data lakes
  • Examine architecture and design choices, and data governance principles and strategies
  • Build a data strategy that scales as your organizational and business needs increase
  • Implement a scalable data lake in the cloud
  • Use cloud-based advanced analytics to gain more value from your data

Publisher resources

View/Submit Errata

Table of contents

  1. Preface
    1. Why I Wrote This Book
    2. Who Should Read This Book?
      1. Introducing Klodars Corporation
    3. Navigating the Book
    4. Conventions Used in This Book
    5. O’Reilly Online Learning
    6. How to Contact Us
    7. Acknowledgments
  2. 1. Big Data—Beyond the Buzz
    1. What Is Big Data?
    2. Elastic Data Infrastructure—The Challenge
    3. Cloud Computing Fundamentals
      1. Cloud Computing Terminology
      2. Value Proposition of the Cloud
    4. Cloud Data Lake Architecture
      1. Limitations of On-Premises Data Warehouse Solutions
      2. What Is a Cloud Data Lake Architecture?
      3. Benefits of a Cloud Data Lake Architecture
    5. Defining Your Cloud Data Lake Journey
    6. Summary
  3. 2. Big Data Architectures on the Cloud
    1. Why Klodars Corporation Moves to the Cloud
    2. Fundamentals of Cloud Data Lake Architectures
      1. A Word on Variety of Data
      2. Cloud Data Lake Storage
      3. Big Data Analytics Engines
      4. Cloud Data Warehouses
    3. Modern Data Warehouse Architecture
      1. Reference Architecture
      2. Sample Use Case for a Modern Data Warehouse Architecture
      3. Benefits and Challenges of Modern Data Warehouse Architecture
    4. Data Lakehouse Architecture
      1. Reference Architecture for the Data Lakehouse
      2. Sample Use Case for Data Lakehouse Architecture
      3. Benefits and Challenges of the Data Lakehouse Architecture
      4. Data Warehouses and Unstructured Data
    5. Data Mesh
      1. Reference Architecture
      2. Sample Use Case for a Data Mesh Architecture
      3. Challenges and Benefits of a Data Mesh Architecture
    6. What Is the Right Architecture for Me?
      1. Know Your Customers
      2. Know Your Business Drivers
      3. Consider Your Growth and Future Scenarios
      4. Design Considerations
      5. Hybrid Approaches
    7. Summary
  4. 3. Design Considerations for Your Data Lake
    1. Setting Up the Cloud Data Lake Infrastructure
      1. Identify Your Goals
      2. Plan Your Architecture and Deliverables
      3. Implement the Cloud Data Lake
      4. Release and Operationalize
    2. Organizing Data in Your Data Lake
      1. A Day in the Life of Data
      2. Data Lake Zones
      3. Organization Mechanisms
    3. Introduction to Data Governance
      1. Actors Involved in Data Governance
      2. Data Classification
      3. Metadata Management, Data Catalog, and Data Sharing
      4. Data Access Management
      5. Data Quality and Observability
      6. Data Governance at Klodars Corporation
      7. Data Governance Wrap-Up
    4. Manage Data Lake Costs
      1. Demystifying Data Lake Costs on the Cloud
      2. Data Lake Cost Strategy
    5. Summary
  5. 4. Scalable Data Lakes
    1. A Sneak Peek into Scalability
      1. What Is Scalability?
      2. Scale in Our Day-to-Day Life
      3. Scalability in Data Lake Architectures
    2. Internals of Data Lake Processing Systems
      1. Data Copy Internals
      2. ELT/ETL Processing Internals
      3. A Note on Other Interactive Queries
    3. Considerations for Scalable Data Lake Solutions
      1. Pick the Right Cloud Offerings
      2. Plan for Peak Capacity
      3. Data Formats and Job Profile
    4. Summary
  6. 5. Optimizing Cloud Data Lake Architectures for Performance
    1. Basics of Measuring Performance
      1. Goals and Metrics for Performance
      2. Measuring Performance
      3. Optimizing for Faster Performance
    2. Cloud Data Lake Performance
      1. SLAs, SLOs, and SLIs
      2. Example: How Klodars Corporation Managed Its SLAs, SLOs, and SLIs
    3. Drivers of Performance
      1. Performance Drivers for a Copy Job
      2. Performance Drivers for a Spark Job
    4. Optimization Principles and Techniques for Performance Tuning
      1. Data Formats
      2. Data Organization and Partitioning
      3. Choosing the Right Configurations on Apache Spark
    5. Minimize Overheads with Data Transfer
    6. Premium Offerings and Performance
      1. The Case of Bigger Virtual Machines
      2. The Case of Flash Storage
    7. Summary
  7. 6. Deep Dive on Data Formats
    1. Why Do We Need These Open Data Formats?
      1. Why Do We Need to Store Tabular Data?
      2. Why Is It a Problem to Store Tabular Data in a Cloud Data Lake Storage?
    2. Delta Lake
      1. Why Was Delta Lake Founded?
      2. How Does Delta Lake Work?
      3. When Do You Use Delta Lake?
    3. Apache Iceberg
      1. Why Was Apache Iceberg Founded?
      2. How Does Apache Iceberg Work?
      3. When Do You Use Apache Iceberg?
    4. Apache Hudi
      1. Why Was Apache Hudi Founded?
      2. How Does Apache Hudi Work?
      3. When Do You Use Apache Hudi?
    5. Summary
  8. 7. Decision Framework for Your Architecture
    1. Cloud Data Lake Assessment
      1. Cloud Data Lake Assessment Questionnaire
    2. Analysis for Your Cloud Data Lake Assessment
      1. Starting from Scratch
      2. Migrating an Existing Data Lake or Data Warehouse to the Cloud
      3. Improving an Existing Cloud Data Lake
    3. Phase 1 of Decision Framework: Assess
      1. Understand Customer Requirements
      2. Understand Opportunities for Improvement
      3. Know Your Business Drivers
      4. Complete the Assess Phase by Prioritizing the Requirements
    4. Phase 2 of Decision Framework: Define
      1. Finalize the Design Choices for the Cloud Data Lake
      2. Plan Your Cloud Data Lake Project Deliverables
    5. Phase 3 of Decision Framework: Implement
    6. Phase 4 of Decision Framework: Operationalize
    7. Summary
  9. 8. Six Lessons for a Data Informed Future
    1. Lesson 1: Focus on the How and When, Not the If and Why, When It Comes to Cloud Data Lakes
    2. Lesson 2: With Great Power Comes Great Responsibility—Data Is No Exception
    3. Lesson 3: Customers Lead Technology, Not the Other Way Around
    4. Lesson 4: Change Is Inevitable, so Be Prepared
    5. Lesson 5: Build Empathy and Prioritize Ruthlessly
    6. Lesson 6: Big Impact Does Not Happen Overnight
    7. Summary
  10. A. Cloud Data Lake Decision Framework Template
    1. Phase 1: Assess Framework
    2. Phase 2: Define Framework
      1. Planning the Cloud Data Lake Deliverables
    3. Phase 3: Implement Framework
  11. Index
  12. About the Author

Product information

  • Title: The Cloud Data Lake
  • Author(s): Rukmani Gopalan
  • Release date: December 2022
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781098116583