book

Data Engineering Best Practices

Name: Data Engineering Best Practices
ISBN: 9781803244983

by Richard J. Schiller, David Larochelle

October 2024

Intermediate to advanced

550 pages

20h 8m

English

Packt Publishing

Read now

Unlock full access

Richard J. Schiller
David LarochelleData Engineering Best Practices
Contributors
About the authors
About the reviewers
Preface
Who this book is forWhat this book coversTo get the most out of this bookGet in touchShare Your ThoughtsDownload a free PDF copy of this book
Chapter 1: Overview of the Business Problem Statement
What is the business problem statement?Anti-patterns to avoidPatterns in the future-proof architectureFuture-proofing is …Organization into zone considerationsCloud limitationsThe Intelligence AgeUse case definitionsThe mission, the vision, and the strategyPrinciples and the development life cycleThe architecture definition, best practices, and key considerationsThe DataOps convergenceSummary
Chapter 2: A Data Engineer’s Journey – Background Challenges
Challenge #1 – platform architectures change rapidlyPlatform architectures in the 21st centuryImpacts on business strategyA flexible software development life cycle to manage platform riskChallenge #2 – Total cost of ownership (TCO) is highETL architecture costs are high!Buy versus build choices impact a solution’s longevityChallenge #3 – Evolving data repository patterns – identifying big rocks for data engineersIntake, integration, and storage challenges in data engineeringIdentifying the big rocks to be placed first into your designBeing able to handle technology hypeSummary
Chapter 3: A Data Engineer’s Journey – IT’s Vision and Mission
The visionDevelop the IT engineering visionVision summaryThe mission and the IT strategyIT’s visionIT’s missionIT mission summaryPrinciples, frameworks, and best practicesThe architecture reflects the visionPrinciples summaryData engineering patterns for IT operabilityWhat patterns are required and how are they specified?Pattern summarySummary
Chapter 4: Architecture Principles
Architecture principles overviewArchitecture foundationData lake, mesh, and fabricData immutabilityThird party tool, cloud platform-as-a-service (PaaS), and framework integrationsData mesh principlesData mesh metadataData semantics in the data meshData mesh, security, and tech stack considerationsWhat are the key foundational takeaways?Architecture principles in depthPrinciple #1 – Data lake as a centerpiece? No, implement the data journey!Principle #2 – A data lake’s immutable data is to remain explorablePrinciple #3 – A data lake’s immutable data remains available for analyticsPrinciple #4 – A data lake’s sources are discoverablePrinciple #5 – A data lake’s tooling should be consistent with the architecturePrinciple #6 – A data mesh defines data to be governed by domain-driven ownershipPrinciple #7 – A data mesh defines the data and derives insights as a productPrinciple #8 – A data mesh defines data, information, and insights to be self-servicePrinciple #9 – A data mesh implements a federated governance processing systemPrinciple #10 – Metadata is associated with datasets and is relevant to the businessPrinciple #11 – Dataset lineage and at-rest metadata is subject to life cycle governancePrinciple #12 – Datasets and metadata require cataloging and discovery servicesPrinciple #13 – Semantic metadata guarantees correct business understanding at all stages in the data journeyPrinciple #14 – Data big rock architecture choices (time series, correction processing, security, privacy, and so on) are to be handled in the design earlyPrinciple #15 – Implement foundational capabilities in the architecture framework firstSummary
Chapter 5: Architecture Framework – Conceptual Architecture Best Practices
Conceptual architecture overviewBest practice organizationHow does the conceptual architecture align with the logical architecture and physical architecture?Conceptual architecture best practicesConceptual architecture descriptionConceptual architecture glossaryWhat are the data architecture’s key issues identified in the conceptual architecture?Best practice composition of the conceptual architectureConceptual to logical architecture mappingSummary
Chapter 6: Architecture Framework – Logical Architecture Best Practices
Logical architecture overviewOrganizing best practicesHow does the logical architecture align with the conceptual and physical architecture?Detailed capabilities of the ingestion zonesETL data pipelinesBronze standard datasetsDetailed capabilities of the transformation zonesData quality featuresData lake house and warehouseGold and silver standard datasetsDetailed capabilities of the consumption zonesData analyticsAccessing silver standard datasets from the consumption zoneTrade-offs between public cloud, on-premises, and multi-cloudCost of ingest or egress for cloud dataCost of a dedicated network line to the point of serviceCost of provisioningCost of monitoring and observabilityHybrid or multi-cloud choices!The benefits of a multi-cloud strategySummary

Chapter 7: Architecture Framework – Physical Architecture Best Practices
Physical architecture overviewBest practice organizationHow does the physical architecture align with the logical and conceptual architecture?How should the physical architecture align with the operational processes/capabilities of the solution?Examples of physical reference architecturesSummary
Chapter 8: Software Engineering Best Practice Considerations
SBP 1 – follow the architecture!The core value of architectural integrityThe downstream impact of deviatingEnsuring adherence in your data engineering teamContinuous evolution and architectureConclusionSBP 2 – implement Agile methodology for your organization!Introduction to Agile methodologyAgile principles and their significance in data engineeringBenefits of implementing Agile in data engineeringChallenges and considerations in Agile data engineeringSteps to implement Agile in data engineeringTools and Agile practices tailored for data engineeringConclusionSBP 3 – generate objectives and key results (OKRs)!Introduction and deep dive into OKRsCrafting data-centric OKRsPotential challenges with OKRs in data engineeringReviewing and iterating on OKRs in a data contextSBP 4 – implement data as a product!SBP 5 – implement shift left testing (SLT) processes!Understanding SLTBenefits of SLT in data engineeringImplementing shift left testingSpecific shift left testing strategies for data engineeringChallenges in shift left testing for data engineeringTools and technologies to facilitate shift left in data engineeringSynergy with other data best practicesSBP 6 – implement the difficult first!The philosophy of tackling the hard tasks firstHow data engineers can prioritize difficult tasksImplementing difficult data tasksSynergy with other data best practicesConclusionSBP 7 – avoid premature optimizationThe true cost of premature optimizationRecognizing and avoiding the trap in data engineeringBalancing performance needs and over-optimization in data engineeringSynergy with other data best practicesSBP 8 – automate cloud code snippet deployments with standard deployment scripted wrappersThe importance of deployment automationThe deployment model choicesBenefits of using scripted deployment wrappersVersion control – ensuring consistency and traceabilityRelevance to data engineering in cloud environmentsPractical implementation stepsChallenges and precautionsSynergy with other software and data best practicesSBP 9 – define and implement NFRs firstDistinguishing functional (FRs) from non-functional requirements (NFRs)Relevance to data engineeringKey NFRs in cloud data engineeringDefining and implementing NFRsRisks of neglecting early implementation of NFRsSBP 10 – implement data journey journaling to facilitate future problem resolutionRelevance to data engineeringChallenges and considerationsSBP 11 – implement data journey pipelines that are experimental first!Enabling data pipeline experimentation as datasets are readiedReleasing data like codeChallenges and considerationsSBP 12 – choose languages with solid reasoningKey languages in data engineering and their rolesThe pressures and limitations imposed by PaaS offeringsPitfalls to avoidSBP 13 – drive scripting and PaaS code with parameterization using a secure configuration management repository toolThe power of parameterization and configuration managementThe growth of configuration complexityWhy parameterize?Configuration management repositories and configuration management databases (CMDBs)Best practices for secure configuration managementSBP 14 – be prepared to prune dead code over timeThe accumulation of dead code in software and PaaS systemsThe unique challenge of PaaS service configurationsPruning dead codeSBP 15 – if it doesn’t fit, don’t force it; use a microservicePaaS and its boundariesMicroservices as a contingency strategyChallenges and considerations of this dual approachPitfalls to avoidSummary
Chapter 9: Key Considerations for Agile SDLC Best Practices
Prevent Agile from being fragileAgile methodologyCore principles of Agile from the Agile ManifestoAgile and data engineeringWhy Agile?The impact of a unified Agile approach on team performance and system qualitySoftware Development Lifecycle (SDLC) processesObjectives and Key Results (OKRs)Agile methodology (tuned to the organization)Business development strategyTest/quality strategyOperational strategyData security strategySummary
Chapter 10: Key Considerations for Quality Testing Best Practices
Quality testing overviewHow not to test!Evolving the test disciplineTest terminologyKey test definitionsTest driven development (TDD)Acceptance test driven development (ATDD) versus developer test driven development (DTDD)Behavioral driven development (BDD)Shift left testingTest framework exampleData wrangling and profilingDeterministic data profilingMachine learning driven data QASummary
Chapter 11: Key Considerations for IT Operational Service Best Practices
IT operational best practices overviewIT operational best practices – introductionService Level Agreements (SLAs)Data contract service level agreements/data contract managementContinuous integration/continuous deployment (CI/CD)Observability with proactive alertingAutomated data and system anomaly detection and remediationData system anomaly detectionApplication Performance MonitoringBlue/green versus continuous deployment trade-offsKey takeawaysSummary
Chapter 12: Key Considerations for Data Service Best Practices
Data service best practices overviewSoftware and data engineering drivers for best practicesZero trust versus defense in depthNational data localizationPrivacy protectionPersonally Identifiable Information (PII)Data service engineering best practicesData engineering best practice 1 – implement a data mesh, not just a data fabricData engineering best practice 2 – implement data pipelines (for analytics)Data engineering best practice 3 – implement data pipelines (for machine learning)Data engineering best practice 4 – use equivalent production and staging environmentsData engineering best practice 5 – a pipeline’s concurrent threads should run and scale in a distributed mannerData engineering best practice 6 – data pipelines should run as streams making use of PaaS services where possibleData engineering best practice 7 – create DataOps standards for data pipeline developmentData engineering best practice 8 – implement tool selection criteria with weighted selection toward built-in integration with core components of the architectureData engineering best practice 9 – implement Pub/Sub models for economy of scale when supporting customers with the same dataset subscriptionData engineering best practice 10 – data wrangling tool selection to create a clean gold zone copy of datasetsData engineering best practice 11 – a data catalog is part of an essential metadata implementationData engineering best practice 12 – define data owners, security, rights, and access for consumers upfrontData engineering best practice 13 – train your experts and let them train and retrain othersData engineering best practice 14 – handle errors gracefullyData engineering best practice 15 – run multiple data pipeline tasks as a directed acyclic graph (DAG)Summary
Chapter 13: Key Considerations for Management Best Practices
Niche focus areas – best practices overviewData profilingHeuristic data analysisGapsCalendar trend deviationsMetadata strategyAzure cloud provider perspective on metadataA metadata tool provider’s perspective on metadataA knowledge engineering provider’s perspective on metadataMetadata perspectives summarySummary
Chapter 14: Key Considerations for Data Delivery Best Practices
Data delivery best practices overviewStreaming data deliveryData delivery with publishing/subscribing (pub/sub) methodsData delivery streaming with examplesConsumable data delivery as a repositoryCustom-built analytics sandbox with Confidential ComputeUsing a third-party aggregator for analyticsData delivery into cloud service provided areasUsing cloud provider’s offeringsBulk data deliveryUsing various cloud provider offeringsSummary
Chapter 15: Other Considerations – Measures, Calculations, Restatements, and Data Science Best Practices
Overview of other data engineering problems for considerationData engineer statement of valueModern data science/analysis workbenchWhat are the capabilities of a modern data science/analyst’s workbench?Using notebooks in productionCalculations and measuresDifficult analytics featuresKey capabilities in the analytics workbenchHistorical data correction processingNotebooksNotebook technologiesSummary
Chapter 16: Machine Learning Pipeline Best Practices and Processes
Machine learning (ML)/artificial intelligence (AI) overviewThe current and future state of AIMachine learning pipelineModel governance and complianceWhy are models rarely deployed?What is the MLOps model life cycle?Technology necessitiesData annotationSampling the data lakeManaging data annotationReal data versus created (synthetic) dataModel trainingModel testing and evaluationSmoke testsML asset deploymentA/B testingHandling regressionsMLOps frameworksSummary
Chapter 17: Takeaway Summary – Putting It All Together
People, technology, and processesWorking with other peopleDevelop your solutions’ technology with clear processesFocus on the process of information value creationOther important challengesBusiness realitiesSecurity and privacyKnowledge engineeringArtificial intelligenceSummary
Chapter 18: Appendix and Use Cases
Use cases overviewTechnology background deep-divePrompt engineering frameworks with examplesLearn about new knowledge base generation tools with examplesHigh level use casesHealth Sciences Knowledge Graph (KG)Life Sciences Semantic Information Engine (ELSSIE)Microblog message analysis (Dataminr)Summary
Index
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your ThoughtsDownload a free PDF copy of this book

Overview

Unlock the secrets to building scalable and efficient data architectures with 'Data Engineering Best Practices.' This book provides in-depth guidance on designing, implementing, and optimizing cloud-based data pipelines. You will gain valuable insights into best practices, agile workflows, and future-proof designs.

What this Book will help me do

Effectively plan and architect scalable data solutions leveraging cloud-first strategies.
Master agile processes tailored to data engineering for improved project outcomes.
Implement secure, efficient, and reliable data pipelines optimized for analytics and AI.
Apply real-world design patterns and avoid common pitfalls in data flow and processing.
Create future-ready data engineering solutions following industry-proven frameworks.

Author(s)

Richard J. Schiller and David Larochelle are seasoned data engineering experts with decades of experience crafting efficient and secure cloud-based infrastructures. Their collaborative writing distills years of real-world expertise into practical advice aimed at helping engineers succeed in a rapidly evolving field.

Who is it for?

This book is ideal for data engineers, ETL specialists, and big data professionals seeking to enhance their knowledge in cloud-based solutions. Some familiarity with data engineering, ETL pipelines, and big data technologies is helpful. It suits those keen on mastering advanced practices, improving agility, and developing efficient data pipelines. Perfect for anyone looking to future-proof their skills in data engineering.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781803244983

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills