book

Data Engineering Best Practices

by Richard J. Schiller, David Larochelle

October 2024

Intermediate to advanced

550 pages

20h 8m

English

Packt Publishing

Read now

Unlock full access

David LarochelleData Engineering Best Practices
About the authors
Who this book is forWhat this book coversTo get the most out of this bookGet in touchShare Your ThoughtsDownload a free PDF copy of this book
What is the business problem statement?Anti-patterns to avoidPatterns in the future-proof architectureFuture-proofing is …Organization into zone considerationsCloud limitationsThe Intelligence AgeUse case definitionsThe mission, the vision, and the strategyPrinciples and the development life cycleThe architecture definition, best practices, and key considerationsThe DataOps convergenceSummary
Challenge #1 – platform architectures change rapidlyPlatform architectures in the 21st centuryImpacts on business strategyA flexible software development life cycle to manage platform riskChallenge #2 – Total cost of ownership (TCO) is highETL architecture costs are high!Buy versus build choices impact a solution’s longevityChallenge #3 – Evolving data repository patterns – identifying big rocks for data engineersIntake, integration, and storage challenges in data engineeringIdentifying the big rocks to be placed first into your designBeing able to handle technology hypeSummary
The visionDevelop the IT engineering visionVision summaryThe mission and the IT strategyIT’s visionIT’s missionIT mission summaryPrinciples, frameworks, and best practicesThe architecture reflects the visionPrinciples summaryData engineering patterns for IT operabilityWhat patterns are required and how are they specified?Pattern summarySummary
Architecture principles overviewArchitecture foundationData lake, mesh, and fabricData immutabilityThird party tool, cloud platform-as-a-service (PaaS), and framework integrationsData mesh principlesData mesh metadataData semantics in the data meshData mesh, security, and tech stack considerationsWhat are the key foundational takeaways?Architecture principles in depthPrinciple #1 – Data lake as a centerpiece? No, implement the data journey!Principle #2 – A data lake’s immutable data is to remain explorablePrinciple #3 – A data lake’s immutable data remains available for analyticsPrinciple #4 – A data lake’s sources are discoverablePrinciple #5 – A data lake’s tooling should be consistent with the architecturePrinciple #6 – A data mesh defines data to be governed by domain-driven ownershipPrinciple #7 – A data mesh defines the data and derives insights as a productPrinciple #8 – A data mesh defines data, information, and insights to be self-servicePrinciple #9 – A data mesh implements a federated governance processing systemPrinciple #10 – Metadata is associated with datasets and is relevant to the businessPrinciple #11 – Dataset lineage and at-rest metadata is subject to life cycle governancePrinciple #12 – Datasets and metadata require cataloging and discovery servicesPrinciple #13 – Semantic metadata guarantees correct business understanding at all stages in the data journeyPrinciple #14 – Data big rock architecture choices (time series, correction processing, security, privacy, and so on) are to be handled in the design earlyPrinciple #15 – Implement foundational capabilities in the architecture framework firstSummary
Conceptual architecture overviewBest practice organizationHow does the conceptual architecture align with the logical architecture and physical architecture?Conceptual architecture best practicesConceptual architecture descriptionConceptual architecture glossaryWhat are the data architecture’s key issues identified in the conceptual architecture?Best practice composition of the conceptual architectureConceptual to logical architecture mappingSummary
Logical architecture overviewOrganizing best practicesHow does the logical architecture align with the conceptual and physical architecture?Detailed capabilities of the ingestion zonesETL data pipelinesBronze standard datasetsDetailed capabilities of the transformation zonesData quality featuresData lake house and warehouseGold and silver standard datasetsDetailed capabilities of the consumption zonesData analyticsAccessing silver standard datasets from the consumption zoneTrade-offs between public cloud, on-premises, and multi-cloudCost of ingest or egress for cloud dataCost of a dedicated network line to the point of serviceCost of provisioningCost of monitoring and observabilityHybrid or multi-cloud choices!The benefits of a multi-cloud strategySummary

Physical architecture overviewBest practice organizationHow does the physical architecture align with the logical and conceptual architecture?How should the physical architecture align with the operational processes/capabilities of the solution?Examples of physical reference architecturesSummary
SBP 1 – follow the architecture!The core value of architectural integrityThe downstream impact of deviatingEnsuring adherence in your data engineering teamContinuous evolution and architectureConclusionSBP 2 – implement Agile methodology for your organization!Introduction to Agile methodologyAgile principles and their significance in data engineeringBenefits of implementing Agile in data engineeringChallenges and considerations in Agile data engineeringSteps to implement Agile in data engineeringTools and Agile practices tailored for data engineeringConclusionSBP 3 – generate objectives and key results (OKRs)!Introduction and deep dive into OKRsCrafting data-centric OKRsPotential challenges with OKRs in data engineeringReviewing and iterating on OKRs in a data contextSBP 4 – implement data as a product!SBP 5 – implement shift left testing (SLT) processes!Understanding SLTBenefits of SLT in data engineeringImplementing shift left testingSpecific shift left testing strategies for data engineeringChallenges in shift left testing for data engineeringTools and technologies to facilitate shift left in data engineeringSynergy with other data best practicesSBP 6 – implement the difficult first!The philosophy of tackling the hard tasks firstHow data engineers can prioritize difficult tasksImplementing difficult data tasksSynergy with other data best practicesConclusionSBP 7 – avoid premature optimizationThe true cost of premature optimizationRecognizing and avoiding the trap in data engineeringBalancing performance needs and over-optimization in data engineeringSynergy with other data best practicesSBP 8 – automate cloud code snippet deployments with standard deployment scripted wrappersThe importance of deployment automationThe deployment model choicesBenefits of using scripted deployment wrappersVersion control – ensuring consistency and traceabilityRelevance to data engineering in cloud environmentsPractical implementation stepsChallenges and precautionsSynergy with other software and data best practicesSBP 9 – define and implement NFRs firstDistinguishing functional (FRs) from non-functional requirements (NFRs)Relevance to data engineeringKey NFRs in cloud data engineeringDefining and implementing NFRsRisks of neglecting early implementation of NFRsSBP 10 – implement data journey journaling to facilitate future problem resolutionRelevance to data engineeringChallenges and considerationsSBP 11 – implement data journey pipelines that are experimental first!Enabling data pipeline experimentation as datasets are readiedReleasing data like codeChallenges and considerationsSBP 12 – choose languages with solid reasoningKey languages in data engineering and their rolesThe pressures and limitations imposed by PaaS offeringsPitfalls to avoidSBP 13 – drive scripting and PaaS code with parameterization using a secure configuration management repository toolThe power of parameterization and configuration managementThe growth of configuration complexityWhy parameterize?Configuration management repositories and configuration management databases (CMDBs)Best practices for secure configuration managementSBP 14 – be prepared to prune dead code over timeThe accumulation of dead code in software and PaaS systemsThe unique challenge of PaaS service configurationsPruning dead codeSBP 15 – if it doesn’t fit, don’t force it; use a microservicePaaS and its boundariesMicroservices as a contingency strategyChallenges and considerations of this dual approachPitfalls to avoidSummary
Prevent Agile from being fragileAgile methodologyCore principles of Agile from the Agile ManifestoAgile and data engineeringWhy Agile?The impact of a unified Agile approach on team performance and system qualitySoftware Development Lifecycle (SDLC) processesObjectives and Key Results (OKRs)Agile methodology (tuned to the organization)Business development strategyTest/quality strategyOperational strategyData security strategySummary
Quality testing overviewHow not to test!Evolving the test disciplineTest terminologyKey test definitionsTest driven development (TDD)Acceptance test driven development (ATDD) versus developer test driven development (DTDD)Behavioral driven development (BDD)Shift left testingTest framework exampleData wrangling and profilingDeterministic data profilingMachine learning driven data QASummary
IT operational best practices overviewIT operational best practices – introductionService Level Agreements (SLAs)Data contract service level agreements/data contract managementContinuous integration/continuous deployment (CI/CD)Observability with proactive alertingAutomated data and system anomaly detection and remediationData system anomaly detectionApplication Performance MonitoringBlue/green versus continuous deployment trade-offsKey takeawaysSummary
Data service best practices overviewSoftware and data engineering drivers for best practicesZero trust versus defense in depthNational data localizationPrivacy protectionPersonally Identifiable Information (PII)Data service engineering best practicesData engineering best practice 1 – implement a data mesh, not just a data fabricData engineering best practice 2 – implement data pipelines (for analytics)Data engineering best practice 3 – implement data pipelines (for machine learning)Data engineering best practice 4 – use equivalent production and staging environmentsData engineering best practice 5 – a pipeline’s concurrent threads should run and scale in a distributed mannerData engineering best practice 6 – data pipelines should run as streams making use of PaaS services where possibleData engineering best practice 7 – create DataOps standards for data pipeline developmentData engineering best practice 8 – implement tool selection criteria with weighted selection toward built-in integration with core components of the architectureData engineering best practice 9 – implement Pub/Sub models for economy of scale when supporting customers with the same dataset subscriptionData engineering best practice 10 – data wrangling tool selection to create a clean gold zone copy of datasetsData engineering best practice 11 – a data catalog is part of an essential metadata implementationData engineering best practice 12 – define data owners, security, rights, and access for consumers upfrontData engineering best practice 13 – train your experts and let them train and retrain othersData engineering best practice 14 – handle errors gracefullyData engineering best practice 15 – run multiple data pipeline tasks as a directed acyclic graph (DAG)Summary
Niche focus areas – best practices overviewData profilingHeuristic data analysisGapsCalendar trend deviationsMetadata strategyAzure cloud provider perspective on metadataA metadata tool provider’s perspective on metadataA knowledge engineering provider’s perspective on metadataMetadata perspectives summarySummary
Data delivery best practices overviewStreaming data deliveryData delivery with publishing/subscribing (pub/sub) methodsData delivery streaming with examplesConsumable data delivery as a repositoryCustom-built analytics sandbox with Confidential ComputeUsing a third-party aggregator for analyticsData delivery into cloud service provided areasUsing cloud provider’s offeringsBulk data deliveryUsing various cloud provider offeringsSummary
Overview of other data engineering problems for considerationData engineer statement of valueModern data science/analysis workbenchWhat are the capabilities of a modern data science/analyst’s workbench?Using notebooks in productionCalculations and measuresDifficult analytics featuresKey capabilities in the analytics workbenchHistorical data correction processingNotebooksNotebook technologiesSummary
Machine learning (ML)/artificial intelligence (AI) overviewThe current and future state of AIMachine learning pipelineModel governance and complianceWhy are models rarely deployed?What is the MLOps model life cycle?Technology necessitiesData annotationSampling the data lakeManaging data annotationReal data versus created (synthetic) dataModel trainingModel testing and evaluationSmoke testsML asset deploymentA/B testingHandling regressionsMLOps frameworksSummary
People, technology, and processesWorking with other peopleDevelop your solutions’ technology with clear processesFocus on the process of information value creationOther important challengesBusiness realitiesSecurity and privacyKnowledge engineeringArtificial intelligenceSummary
Use cases overviewTechnology background deep-divePrompt engineering frameworks with examplesLearn about new knowledge base generation tools with examplesHigh level use casesHealth Sciences Knowledge Graph (KG)Life Sciences Semantic Information Engine (ELSSIE)Microblog message analysis (Dataminr)Summary
Other Books You May EnjoyPackt is searching for authors like youShare Your ThoughtsDownload a free PDF copy of this book

Content preview from Data Engineering Best Practices

2 A Data Engineer’s Journey – Background Challenges

The purpose of this chapter is to explain the challenges you must face to navigate a successful journey as a data engineer. The intricate nature of managing vast data, evolving technologies, and ensuring efficient data pipelines are some of the hurdles that make data engineering a demanding field. In this chapter, we will explain why data engineering is hard, and provide a foundation for the development of principles to be discussed in later chapters. We will also provide insight when navigating these challenges. Finally, we will provide an overview of the data engineering approaches you can use to scope out the current and near-future technology landscape.

We will discuss three of the main ...