book

Simplifying Data Engineering and Analytics with Delta

by Anindita Mahapatra

July 2022

Beginner to intermediate

334 pages

7h 33m

English

Packt Publishing

Read now

Unlock full access

ForewordContributorsAbout the authorAbout the reviewer
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchShare Your Thoughts
The motivation behind data engineeringUse casesHow big is big data?But isn't ML and AI all the rage today?Understanding the role of data personasBig data ecosystemWhat characterizes big data?Classifying dataReaping value from dataTop challenges of big data systemsEvolution of data systemsRise of cloud data platformsSQL and NoSQL systemsOLTP and OLAP systemsData platform service models Distributed computingSMP and MPP computingParallel and distributed computingHadoopSparkHadoop versus SparkBusiness justification for tech spendingStrategy for business transformation to use data as an assetBig data trends and best practicesSummary
Technical requirementsWhat is data modeling and why should you care?Advantages of a data modeling exerciseStages of data modelingData modeling approaches for different data storesRelational data modelingNon-relational data modelingUnderstanding metadata – data about dataData catalogTypes of metadataWhy is metadata management the nerve center of data?Moving and transforming data using ETLScenarios to consider for building ETL pipelinesPeriodic and continuous ingestionBulk data migration Change data capture Slowly changing dimensionsJob orchestrationHow to choose the right data formatText format versus binary formatRow versus column formatsWhen to use which formatLeveraging data compressionCommon big data design patternsIngestionUnified APISpeed layerTransformationsHandling schema changes ACID transactions Multihop pipeline PersistSeparation of compute from storageMultiple destinationsDenormalizationIn-stream analyticsBest practicesSummaryFurther reading
Technical requirementsMotivation for DeltaA case of too many is too littleData silos to data swampsCharacteristics of curated data lakesDDL commandsCREATEDML commandsAPPENDUPDATEDELETE MERGE Demystifying DeltaFormat layout on diskThe main features of DeltaACID transaction supportSchema evolutionUnifying batch and streaming workloadsTime travelPerformanceData skippingZ-Order clusteringDelta cacheLife with and without DeltaLakehouseCharacteristics of a LakehouseSummary
Technical requirementsMoving toward real-time systemsStreaming conceptsLambda versus Kappa architecturesStreaming ETLExtract – file-based versus event-based streamingTransforming – stream processingLoading – persisting the streamHandling streaming scenariosJoining with other static and dynamic datasetsRecovering from failuresHandling late-arriving dataStateless and stateful in-stream operationsTrade-offs in designing streaming architecturesCost trade-offsHandling latency trade-offsData reprocessingMulti-tenancyDe-duplicationStreaming best practicesSummary
Technical requirementsWhy consolidate disparate data types?Delta unifies all types of dataStructured dataSemi-structured dataUnstructured dataAvoiding patches of data darknessAddressing problems in flight status using DeltaAugmenting domain knowledge constraints to quality Continuous quality monitoringCurating data in stages for analyticsRDD, DataFrames, and datasetsSpark transformations and actionsSpark APIs and UDFsEase of extending to existing and new use casesDelta Lake connectorsSpecialized Delta Lakes by industryHealthcare and life sciences Delta LakeIndustry 4.0 manufacturing Delta LakeFinancial services Delta LakeRetail Delta LakeData governanceGDPR and CCPA complianceRole-based data accessSummary
Technical requirementsUnderstanding use case requirements Minimizing data movement with Delta time travelDelta cloningHandling CDCCDCChange Data Feed (CDF)Handling Slowly Changing Dimensions (SCD)SCD Type 1SCD Type 2Summary

Technical requirementsChoosing the right architecture Understanding what a data warehouse really solvesLacunas of data warehousesDiscovering when a data lake does not sufficeAddressing concurrency and latency requirements with DeltaVisualizing data using BI reportingCan cubes be constructed with Delta?Analyzing tradeoffs in a push versus pull data flowWhy is being open such a big deal?Considerations around data governanceThe rise of the lakehouse categorySummary
Technical requirementsEmphasizing the importance of exploratory data analysis (EDA)From big data to good dataData profilingStatistical analysisApplying sampling techniques to address class imbalance How to detect and address imbalanceSynthetic data generation to deal with data imbalanceAddressing data skewProviding data anonymityHandling bias and variance in dataBias versus varianceHow do we detect bias and variance?How do we fix bias and variance?Compensating for missing and out-of-range dataMonitoring data driftSummary
Technical requirementsData science versus machine learningChallenges of ML developmentFormalizing the ML development processWhat is a model?What is MLOps?Aspirations of a modern ML platformThe role of Delta in an ML pipelineDelta-backed feature storeDelta-backed model trainingDelta-backed model inferencingModel monitoring with DeltaFrom business problem to insight generationSummary
Technical requirementsDaaS The need for data democratization Delta for unstructured dataNLP data (text and audio)Image and video dataData mashups using DeltaData blendingData harmonizationFederated queryFacilitating data sharing with DeltaSetting up Delta sharingBenefits of Delta sharingData clean roomSummary
Technical requirementsWhy operationalize?Understanding and monitoring SLAsScaling and high availabilityPlanning for DR How to decide on the correct DR strategyHow Delta helps with DRGuaranteeing data qualityAutomation of CI/CD pipelines Code under version controlInfrastructure as Code (IaC)Unit and integration testingData as code – An intelligent pipelineSummary
Technical requirementsImproving performance with common strategies Where to look and what to look forOptimizing with DeltaChanging the data layout in storageOther platform optimizationsAutomationIs cost always inversely proportional to performance?Best practices for managing performanceSummary
Provisioning a multi-tenant infrastructure Data democratization via policies and processesCapacity planningManaging and monitoring Data sharingData migrationCOE best practices Summary
Other Books You May EnjoyPackt is searching for authors like youShare Your Thoughts

Overview

This book will guide you through mastering Delta, a robust and versatile protocol for data engineering and analytics. You'll discover how Delta simplifies data workflows, supports both batch and streaming data, and is optimized for analytics applications in various industries. By the end, you will know how to create high-performing, analytics-ready data pipelines.

What this Book will help me do

Understand Delta's unique offering for unifying batch and streaming data processing.
Learn approaches to address data governance, reliability, and scalability challenges.
Gain technical expertise in building data pipelines optimized for analytics and machine learning use.
Master core concepts like data modeling, distributed computing, and Delta's schema evolution features.
Develop and deploy production-grade data engineering solutions leveraging Delta for business intelligence.

Author(s)

Anindita Mahapatra is an experienced data engineer and author with years of expertise in working on Delta and data-driven solutions. Her hands-on approach to explaining complex data concepts makes this book an invaluable resource for professionals in data engineering and analytics.

Who is it for?

Ideal for data engineers, data analysts, and anyone involved in AI/BI workflows, this book suits learners with some basic knowledge of SQL and Python. Whether you're an experienced professional or looking to upgrade your skills with Delta, this book will provide practical insights and actionable knowledge.