book

Simplifying Data Engineering and Analytics with Delta

Name: Simplifying Data Engineering and Analytics with Delta
Author: Anindita Mahapatra
ISBN: 9781801814867

by Anindita Mahapatra

July 2022

Beginner to intermediate

334 pages

7h 33m

English

Packt Publishing

Read now

Unlock full access

Simplifying Data Engineering and Analytics with Delta
ForewordContributorsAbout the authorAbout the reviewer
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchShare Your Thoughts
Section 1 – Introduction to Delta Lake and Data Engineering Principles
Chapter 1: Introduction to Data Engineering
The motivation behind data engineeringUse casesHow big is big data?But isn't ML and AI all the rage today?Understanding the role of data personasBig data ecosystemWhat characterizes big data?Classifying dataReaping value from dataTop challenges of big data systemsEvolution of data systemsRise of cloud data platformsSQL and NoSQL systemsOLTP and OLAP systemsData platform service models Distributed computingSMP and MPP computingParallel and distributed computingHadoopSparkHadoop versus SparkBusiness justification for tech spendingStrategy for business transformation to use data as an assetBig data trends and best practicesSummary
Chapter 2: Data Modeling and ETL
Technical requirementsWhat is data modeling and why should you care?Advantages of a data modeling exerciseStages of data modelingData modeling approaches for different data storesRelational data modelingNon-relational data modelingUnderstanding metadata – data about dataData catalogTypes of metadataWhy is metadata management the nerve center of data?Moving and transforming data using ETLScenarios to consider for building ETL pipelinesPeriodic and continuous ingestionBulk data migration Change data capture Slowly changing dimensionsJob orchestrationHow to choose the right data formatText format versus binary formatRow versus column formatsWhen to use which formatLeveraging data compressionCommon big data design patternsIngestionUnified APISpeed layerTransformationsHandling schema changes ACID transactions Multihop pipeline PersistSeparation of compute from storageMultiple destinationsDenormalizationIn-stream analyticsBest practicesSummaryFurther reading
Chapter 3: Delta – The Foundation Block for Big Data
Technical requirementsMotivation for DeltaA case of too many is too littleData silos to data swampsCharacteristics of curated data lakesDDL commandsCREATEDML commandsAPPENDUPDATEDELETE MERGE Demystifying DeltaFormat layout on diskThe main features of DeltaACID transaction supportSchema evolutionUnifying batch and streaming workloadsTime travelPerformanceData skippingZ-Order clusteringDelta cacheLife with and without DeltaLakehouseCharacteristics of a LakehouseSummary
Section 2 – End-to-End Process of Building Delta Pipelines
Chapter 4: Unifying Batch and Streaming with Delta
Technical requirementsMoving toward real-time systemsStreaming conceptsLambda versus Kappa architecturesStreaming ETLExtract – file-based versus event-based streamingTransforming – stream processingLoading – persisting the streamHandling streaming scenariosJoining with other static and dynamic datasetsRecovering from failuresHandling late-arriving dataStateless and stateful in-stream operationsTrade-offs in designing streaming architecturesCost trade-offsHandling latency trade-offsData reprocessingMulti-tenancyDe-duplicationStreaming best practicesSummary
Chapter 5: Data Consolidation in Delta Lake
Technical requirementsWhy consolidate disparate data types?Delta unifies all types of dataStructured dataSemi-structured dataUnstructured dataAvoiding patches of data darknessAddressing problems in flight status using DeltaAugmenting domain knowledge constraints to quality Continuous quality monitoringCurating data in stages for analyticsRDD, DataFrames, and datasetsSpark transformations and actionsSpark APIs and UDFsEase of extending to existing and new use casesDelta Lake connectorsSpecialized Delta Lakes by industryHealthcare and life sciences Delta LakeIndustry 4.0 manufacturing Delta LakeFinancial services Delta LakeRetail Delta LakeData governanceGDPR and CCPA complianceRole-based data accessSummary
Chapter 6: Solving Common Data Pattern Scenarios with Delta
Technical requirementsUnderstanding use case requirements Minimizing data movement with Delta time travelDelta cloningHandling CDCCDCChange Data Feed (CDF)Handling Slowly Changing Dimensions (SCD)SCD Type 1SCD Type 2Summary

Chapter 7: Delta for Data Warehouse Use Cases
Technical requirementsChoosing the right architecture Understanding what a data warehouse really solvesLacunas of data warehousesDiscovering when a data lake does not sufficeAddressing concurrency and latency requirements with DeltaVisualizing data using BI reportingCan cubes be constructed with Delta?Analyzing tradeoffs in a push versus pull data flowWhy is being open such a big deal?Considerations around data governanceThe rise of the lakehouse categorySummary
Chapter 8: Handling Atypical Data Scenarios with Delta
Technical requirementsEmphasizing the importance of exploratory data analysis (EDA)From big data to good dataData profilingStatistical analysisApplying sampling techniques to address class imbalance How to detect and address imbalanceSynthetic data generation to deal with data imbalanceAddressing data skewProviding data anonymityHandling bias and variance in dataBias versus varianceHow do we detect bias and variance?How do we fix bias and variance?Compensating for missing and out-of-range dataMonitoring data driftSummary
Chapter 9: Delta for Reproducible Machine Learning Pipelines
Technical requirementsData science versus machine learningChallenges of ML developmentFormalizing the ML development processWhat is a model?What is MLOps?Aspirations of a modern ML platformThe role of Delta in an ML pipelineDelta-backed feature storeDelta-backed model trainingDelta-backed model inferencingModel monitoring with DeltaFrom business problem to insight generationSummary
Chapter 10: Delta for Data Products and Services
Technical requirementsDaaS The need for data democratization Delta for unstructured dataNLP data (text and audio)Image and video dataData mashups using DeltaData blendingData harmonizationFederated queryFacilitating data sharing with DeltaSetting up Delta sharingBenefits of Delta sharingData clean roomSummary
Section 3 – Operationalizing and Productionalizing Delta Pipelines
Chapter 11: Operationalizing Data and ML Pipelines
Technical requirementsWhy operationalize?Understanding and monitoring SLAsScaling and high availabilityPlanning for DR How to decide on the correct DR strategyHow Delta helps with DRGuaranteeing data qualityAutomation of CI/CD pipelines Code under version controlInfrastructure as Code (IaC)Unit and integration testingData as code – An intelligent pipelineSummary
Chapter 12: Optimizing Cost and Performance with Delta
Technical requirementsImproving performance with common strategies Where to look and what to look forOptimizing with DeltaChanging the data layout in storageOther platform optimizationsAutomationIs cost always inversely proportional to performance?Best practices for managing performanceSummary
Chapter 13: Managing Your Data Journey
Provisioning a multi-tenant infrastructure Data democratization via policies and processesCapacity planningManaging and monitoring Data sharingData migrationCOE best practices Summary
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your Thoughts

Content preview from Simplifying Data Engineering and Analytics with Delta

Chapter 9: Delta for Reproducible Machine Learning Pipelines

"Repetition is the mother of learning, the father of action, which makes it the architect of accomplishment."

– Zig Ziglar, American author and motivational speaker

In previous chapters, we established the pivotal nature of Delta in architecting data pipelines. What about Machine Learning (ML) pipelines? They involve different personas with different skills and needs. ML has been around for a while; what has changed lately is broad access to large datasets and affordable compute, which has now made it possible for everyone to tinker with ML. Can Delta stand the litmus test of building a reproducible ML pipeline just as effectively as a data pipeline? There are specific challenges ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Productive and Efficient Data Science with Python: With Modularizing, Memory profiles, and Parallel/GPU Processing

Publisher Resources

ISBN: 9781801814867

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Simplifying Data Engineering and Analytics with Delta

by Anindita Mahapatra

Chapter 9: Delta for Reproducible Machine Learning Pipelines

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.