book

Fundamentals of Data Engineering

by Joe Reis, Matt Housley

June 2022

Beginner

450 pages

13h 34m

English

O'Reilly Media, Inc.

Audiobook available

Read now

Unlock full access

Includes

Quizzes

Preface
What This Book Isn’tWhat This Book Is AboutWho Should Read This BookPrerequisitesWhat You’ll Learn and How It Will Improve Your AbilitiesNavigating This BookConventions Used in This BookHow to Contact UsAcknowledgments
I. Foundation and Building Blocks
1. Data Engineering Described
What Is Data Engineering?Data Engineering DefinedThe Data Engineering LifecycleEvolution of the Data EngineerData Engineering and Data ScienceData Engineering Skills and ActivitiesData Maturity and the Data EngineerThe Background and Skills of a Data EngineerBusiness ResponsibilitiesTechnical ResponsibilitiesThe Continuum of Data Engineering Roles, from A to BData Engineers Inside an OrganizationInternal-Facing Versus External-Facing Data EngineersData Engineers and Other Technical RolesData Engineers and Business LeadershipConclusionAdditional Resources
2. The Data Engineering Lifecycle
What Is the Data Engineering Lifecycle?The Data Lifecycle Versus the Data Engineering LifecycleGeneration: Source SystemsStorageIngestionTransformationServing DataMajor Undercurrents Across the Data Engineering LifecycleSecurityData ManagementDataOpsData ArchitectureOrchestrationSoftware EngineeringConclusionAdditional Resources
3. Designing Good Data Architecture
What Is Data Architecture?Enterprise Architecture DefinedData Architecture Defined“Good” Data ArchitecturePrinciples of Good Data ArchitecturePrinciple 1: Choose Common Components WiselyPrinciple 2: Plan for FailurePrinciple 3: Architect for ScalabilityPrinciple 4: Architecture Is LeadershipPrinciple 5: Always Be ArchitectingPrinciple 6: Build Loosely Coupled SystemsPrinciple 7: Make Reversible DecisionsPrinciple 8: Prioritize SecurityPrinciple 9: Embrace FinOpsMajor Architecture ConceptsDomains and ServicesDistributed Systems, Scalability, and Designing for FailureTight Versus Loose Coupling: Tiers, Monoliths, and MicroservicesUser Access: Single Versus MultitenantEvent-Driven ArchitectureBrownfield Versus Greenfield ProjectsExamples and Types of Data ArchitectureData WarehouseData LakeConvergence, Next-Generation Data Lakes, and the Data PlatformModern Data StackLambda ArchitectureKappa ArchitectureThe Dataflow Model and Unified Batch and StreamingArchitecture for IoTData MeshOther Data Architecture ExamplesWho’s Involved with Designing a Data Architecture?ConclusionAdditional Resources
4. Choosing Technologies Across the Data Engineering Lifecycle
Team Size and CapabilitiesSpeed to MarketInteroperabilityCost Optimization and Business ValueTotal Cost of OwnershipTotal Opportunity Cost of OwnershipFinOpsToday Versus the Future: Immutable Versus Transitory TechnologiesOur AdviceLocationOn PremisesCloudHybrid CloudMulticloudDecentralized: Blockchain and the EdgeOur AdviceCloud Repatriation ArgumentsBuild Versus BuyOpen Source SoftwareProprietary Walled GardensOur AdviceMonolith Versus ModularMonolithModularityThe Distributed Monolith PatternOur AdviceServerless Versus ServersServerlessContainersHow to Evaluate Server Versus ServerlessOur AdviceOptimization, Performance, and the Benchmark WarsBig Data...for the 1990sNonsensical Cost ComparisonsAsymmetric OptimizationCaveat EmptorUndercurrents and Their Impacts on Choosing TechnologiesData ManagementDataOpsData ArchitectureOrchestration Example: AirflowSoftware EngineeringConclusionAdditional Resources
II. The Data Engineering Lifecycle in Depth
5. Data Generation in Source Systems
Sources of Data: How Is Data Created?Source Systems: Main IdeasFiles and Unstructured DataAPIsApplication Databases (OLTP Systems)Online Analytical Processing SystemChange Data CaptureLogsDatabase LogsCRUDInsert-OnlyMessages and StreamsTypes of TimeSource System Practical DetailsDatabasesAPIsData SharingThird-Party Data SourcesMessage Queues and Event-Streaming PlatformsWhom You’ll Work WithUndercurrents and Their Impact on Source SystemsSecurityData ManagementDataOpsData ArchitectureOrchestrationSoftware EngineeringConclusionAdditional Resources
6. Storage
Raw Ingredients of Data StorageMagnetic Disk DriveSolid-State DriveRandom Access MemoryNetworking and CPUSerializationCompressionCachingData Storage SystemsSingle Machine Versus Distributed StorageEventual Versus Strong ConsistencyFile StorageBlock StorageObject StorageCache and Memory-Based Storage SystemsThe Hadoop Distributed File SystemStreaming StorageIndexes, Partitioning, and ClusteringData Engineering Storage AbstractionsThe Data WarehouseThe Data LakeThe Data LakehouseData PlatformsStream-to-Batch Storage ArchitectureBig Ideas and Trends in StorageData CatalogData SharingSchemaSeparation of Compute from StorageData Storage Lifecycle and Data RetentionSingle-Tenant Versus Multitenant StorageWhom You’ll Work WithUndercurrentsSecurityData ManagementDataOpsData ArchitectureOrchestrationSoftware EngineeringConclusionAdditional Resources
7. Ingestion
What Is Data Ingestion?Key Engineering Considerations for the Ingestion PhaseBounded Versus Unbounded DataFrequencySynchronous Versus Asynchronous IngestionSerialization and DeserializationThroughput and ScalabilityReliability and DurabilityPayloadPush Versus Pull Versus Poll PatternsBatch Ingestion ConsiderationsSnapshot or Differential ExtractionFile-Based Export and IngestionETL Versus ELTInserts, Updates, and Batch SizeData MigrationMessage and Stream Ingestion ConsiderationsSchema EvolutionLate-Arriving DataOrdering and Multiple DeliveryReplayTime to LiveMessage SizeError Handling and Dead-Letter QueuesConsumer Pull and PushLocationWays to Ingest DataDirect Database ConnectionChange Data CaptureAPIsMessage Queues and Event-Streaming PlatformsManaged Data ConnectorsMoving Data with Object StorageEDIDatabases and File ExportPractical Issues with Common File FormatsShellSSHSFTP and SCPWebhooksWeb InterfaceWeb ScrapingTransfer Appliances for Data MigrationData SharingWhom You’ll Work WithUpstream StakeholdersDownstream StakeholdersUndercurrentsSecurityData ManagementDataOpsOrchestrationSoftware EngineeringConclusionAdditional Resources

8. Queries, Modeling, and Transformation
QueriesWhat Is a Query?The Life of a QueryThe Query OptimizerImproving Query PerformanceQueries on Streaming DataData ModelingWhat Is a Data Model?Conceptual, Logical, and Physical Data ModelsNormalizationTechniques for Modeling Batch Analytical DataModeling Streaming DataTransformationsBatch TransformationsMaterialized Views, Federation, and Query VirtualizationStreaming Transformations and ProcessingWhom You’ll Work WithUpstream StakeholdersDownstream StakeholdersUndercurrentsSecurityData ManagementDataOpsData ArchitectureOrchestrationSoftware EngineeringConclusionAdditional Resources
9. Serving Data for Analytics, Machine Learning, and Reverse ETL
General Considerations for Serving DataTrustWhat’s the Use Case, and Who’s the User?Data ProductsSelf-Service or Not?Data Definitions and LogicData MeshAnalyticsBusiness AnalyticsOperational AnalyticsEmbedded AnalyticsMachine LearningWhat a Data Engineer Should Know About MLWays to Serve Data for Analytics and MLFile ExchangeDatabasesStreaming SystemsQuery FederationData SharingSemantic and Metrics LayersServing Data in NotebooksReverse ETLWhom You’ll Work WithUndercurrentsSecurityData ManagementDataOpsData ArchitectureOrchestrationSoftware EngineeringConclusionAdditional Resources
III. Security, Privacy, and the Future of Data Engineering
10. Security and Privacy
PeopleThe Power of Negative ThinkingAlways Be ParanoidProcessesSecurity Theater Versus Security HabitActive SecurityThe Principle of Least PrivilegeShared Responsibility in the CloudAlways Back Up Your DataAn Example Security PolicyTechnologyPatch and Update SystemsEncryptionLogging, Monitoring, and AlertingNetwork AccessSecurity for Low-Level Data EngineeringConclusionAdditional Resources
11. The Future of Data Engineering
The Data Engineering Lifecycle Isn’t Going AwayThe Decline of Complexity and the Rise of Easy-to-Use Data ToolsThe Cloud-Scale Data OS and Improved Interoperability“Enterprisey” Data EngineeringTitles and Responsibilities Will Morph...Moving Beyond the Modern Data Stack, Toward the Live Data StackThe Live Data StackStreaming Pipelines and Real-Time Analytical DatabasesThe Fusion of Data with ApplicationsThe Tight Feedback Between Applications and MLDark Matter Data and the Rise of...Spreadsheets?!Conclusion
A. Serialization and Compression Technical Details
Serialization FormatsRow-Based SerializationColumnar SerializationHybrid SerializationDatabase Storage EnginesCompression: gzip, bzip2, Snappy, Etc.
B. Cloud Networking
Cloud Network TopologyData Egress ChargesAvailability ZonesRegionsGCP-Specific Networking and Multiregional RedundancyDirect Network Connections to the CloudsCDNsThe Future of Data Egress Fees
Index
About the Authors

Content preview from Fundamentals of Data Engineering

About the Authors

Joe Reis is a business-minded data nerd who’s worked in the data industry for 20 years, with responsibilities ranging from statistical modeling, forecasting, machine learning, data engineering, data architecture, and almost everything else in between. Joe is the CEO and cofounder of Ternary Data, a data engineering and architecture consulting firm based in Salt Lake City, Utah. In addition, he volunteers with several technology groups and teaches at the University of Utah. In his spare time, Joe likes to rock climb, produce electronic music, and take his kids on crazy adventures.

Matt Housley is a data engineering consultant and cloud specialist. After some early programming experience with Logo, Basic, and 6502 assembly, he completed a PhD in mathematics at the University of Utah. Matt then began working in data science, eventually specializing in cloud-based data engineering. He cofounded Ternary Data with Joe Reis, where he leverages his teaching experience to train future data engineers and advise teams on robust data architecture. Matt and Joe also pontificate on all things data on The Monday Morning Data Chat.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781098108298Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Fundamentals of Data Engineering

by Joe Reis, Matt Housley

About the Authors

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.