book

Fundamentals of Data Engineering

by Joe Reis, Matt Housley

June 2022

Beginner

450 pages

13h 34m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Includes

Includes Quizzes

What This Book Isn’tWhat This Book Is AboutWho Should Read This BookPrerequisitesWhat You’ll Learn and How It Will Improve Your AbilitiesNavigating This BookConventions Used in This BookHow to Contact UsAcknowledgments
What Is Data Engineering?Data Engineering DefinedThe Data Engineering LifecycleEvolution of the Data EngineerData Engineering and Data ScienceData Engineering Skills and ActivitiesData Maturity and the Data EngineerThe Background and Skills of a Data EngineerBusiness ResponsibilitiesTechnical ResponsibilitiesThe Continuum of Data Engineering Roles, from A to BData Engineers Inside an OrganizationInternal-Facing Versus External-Facing Data EngineersData Engineers and Other Technical RolesData Engineers and Business LeadershipConclusionAdditional Resources
What Is the Data Engineering Lifecycle?The Data Lifecycle Versus the Data Engineering LifecycleGeneration: Source SystemsStorageIngestionTransformationServing DataMajor Undercurrents Across the Data Engineering LifecycleSecurityData ManagementDataOpsData ArchitectureOrchestrationSoftware EngineeringConclusionAdditional Resources
What Is Data Architecture?Enterprise Architecture DefinedData Architecture Defined“Good” Data ArchitecturePrinciples of Good Data ArchitecturePrinciple 1: Choose Common Components WiselyPrinciple 2: Plan for FailurePrinciple 3: Architect for ScalabilityPrinciple 4: Architecture Is LeadershipPrinciple 5: Always Be ArchitectingPrinciple 6: Build Loosely Coupled SystemsPrinciple 7: Make Reversible DecisionsPrinciple 8: Prioritize SecurityPrinciple 9: Embrace FinOpsMajor Architecture ConceptsDomains and ServicesDistributed Systems, Scalability, and Designing for FailureTight Versus Loose Coupling: Tiers, Monoliths, and MicroservicesUser Access: Single Versus MultitenantEvent-Driven ArchitectureBrownfield Versus Greenfield ProjectsExamples and Types of Data ArchitectureData WarehouseData LakeConvergence, Next-Generation Data Lakes, and the Data PlatformModern Data StackLambda ArchitectureKappa ArchitectureThe Dataflow Model and Unified Batch and StreamingArchitecture for IoTData MeshOther Data Architecture ExamplesWho’s Involved with Designing a Data Architecture?ConclusionAdditional Resources
Team Size and CapabilitiesSpeed to MarketInteroperabilityCost Optimization and Business ValueTotal Cost of OwnershipTotal Opportunity Cost of OwnershipFinOpsToday Versus the Future: Immutable Versus Transitory TechnologiesOur AdviceLocationOn PremisesCloudHybrid CloudMulticloudDecentralized: Blockchain and the EdgeOur AdviceCloud Repatriation ArgumentsBuild Versus BuyOpen Source SoftwareProprietary Walled GardensOur AdviceMonolith Versus ModularMonolithModularityThe Distributed Monolith PatternOur AdviceServerless Versus ServersServerlessContainersHow to Evaluate Server Versus ServerlessOur AdviceOptimization, Performance, and the Benchmark WarsBig Data...for the 1990sNonsensical Cost ComparisonsAsymmetric OptimizationCaveat EmptorUndercurrents and Their Impacts on Choosing TechnologiesData ManagementDataOpsData ArchitectureOrchestration Example: AirflowSoftware EngineeringConclusionAdditional Resources
Sources of Data: How Is Data Created?Source Systems: Main IdeasFiles and Unstructured DataAPIsApplication Databases (OLTP Systems)Online Analytical Processing SystemChange Data CaptureLogsDatabase LogsCRUDInsert-OnlyMessages and StreamsTypes of TimeSource System Practical DetailsDatabasesAPIsData SharingThird-Party Data SourcesMessage Queues and Event-Streaming PlatformsWhom You’ll Work WithUndercurrents and Their Impact on Source SystemsSecurityData ManagementDataOpsData ArchitectureOrchestrationSoftware EngineeringConclusionAdditional Resources
Raw Ingredients of Data StorageMagnetic Disk DriveSolid-State DriveRandom Access MemoryNetworking and CPUSerializationCompressionCachingData Storage SystemsSingle Machine Versus Distributed StorageEventual Versus Strong ConsistencyFile StorageBlock StorageObject StorageCache and Memory-Based Storage SystemsThe Hadoop Distributed File SystemStreaming StorageIndexes, Partitioning, and ClusteringData Engineering Storage AbstractionsThe Data WarehouseThe Data LakeThe Data LakehouseData PlatformsStream-to-Batch Storage ArchitectureBig Ideas and Trends in StorageData CatalogData SharingSchemaSeparation of Compute from StorageData Storage Lifecycle and Data RetentionSingle-Tenant Versus Multitenant StorageWhom You’ll Work WithUndercurrentsSecurityData ManagementDataOpsData ArchitectureOrchestrationSoftware EngineeringConclusionAdditional Resources
What Is Data Ingestion?Key Engineering Considerations for the Ingestion PhaseBounded Versus Unbounded DataFrequencySynchronous Versus Asynchronous IngestionSerialization and DeserializationThroughput and ScalabilityReliability and DurabilityPayloadPush Versus Pull Versus Poll PatternsBatch Ingestion ConsiderationsSnapshot or Differential ExtractionFile-Based Export and IngestionETL Versus ELTInserts, Updates, and Batch SizeData MigrationMessage and Stream Ingestion ConsiderationsSchema EvolutionLate-Arriving DataOrdering and Multiple DeliveryReplayTime to LiveMessage SizeError Handling and Dead-Letter QueuesConsumer Pull and PushLocationWays to Ingest DataDirect Database ConnectionChange Data CaptureAPIsMessage Queues and Event-Streaming PlatformsManaged Data ConnectorsMoving Data with Object StorageEDIDatabases and File ExportPractical Issues with Common File FormatsShellSSHSFTP and SCPWebhooksWeb InterfaceWeb ScrapingTransfer Appliances for Data MigrationData SharingWhom You’ll Work WithUpstream StakeholdersDownstream StakeholdersUndercurrentsSecurityData ManagementDataOpsOrchestrationSoftware EngineeringConclusionAdditional Resources

QueriesWhat Is a Query?The Life of a QueryThe Query OptimizerImproving Query PerformanceQueries on Streaming DataData ModelingWhat Is a Data Model?Conceptual, Logical, and Physical Data ModelsNormalizationTechniques for Modeling Batch Analytical DataModeling Streaming DataTransformationsBatch TransformationsMaterialized Views, Federation, and Query VirtualizationStreaming Transformations and ProcessingWhom You’ll Work WithUpstream StakeholdersDownstream StakeholdersUndercurrentsSecurityData ManagementDataOpsData ArchitectureOrchestrationSoftware EngineeringConclusionAdditional Resources
General Considerations for Serving DataTrustWhat’s the Use Case, and Who’s the User?Data ProductsSelf-Service or Not?Data Definitions and LogicData MeshAnalyticsBusiness AnalyticsOperational AnalyticsEmbedded AnalyticsMachine LearningWhat a Data Engineer Should Know About MLWays to Serve Data for Analytics and MLFile ExchangeDatabasesStreaming SystemsQuery FederationData SharingSemantic and Metrics LayersServing Data in NotebooksReverse ETLWhom You’ll Work WithUndercurrentsSecurityData ManagementDataOpsData ArchitectureOrchestrationSoftware EngineeringConclusionAdditional Resources
PeopleThe Power of Negative ThinkingAlways Be ParanoidProcessesSecurity Theater Versus Security HabitActive SecurityThe Principle of Least PrivilegeShared Responsibility in the CloudAlways Back Up Your DataAn Example Security PolicyTechnologyPatch and Update SystemsEncryptionLogging, Monitoring, and AlertingNetwork AccessSecurity for Low-Level Data EngineeringConclusionAdditional Resources
The Data Engineering Lifecycle Isn’t Going AwayThe Decline of Complexity and the Rise of Easy-to-Use Data ToolsThe Cloud-Scale Data OS and Improved Interoperability“Enterprisey” Data EngineeringTitles and Responsibilities Will Morph...Moving Beyond the Modern Data Stack, Toward the Live Data StackThe Live Data StackStreaming Pipelines and Real-Time Analytical DatabasesThe Fusion of Data with ApplicationsThe Tight Feedback Between Applications and MLDark Matter Data and the Rise of...Spreadsheets?!Conclusion
Serialization FormatsRow-Based SerializationColumnar SerializationHybrid SerializationDatabase Storage EnginesCompression: gzip, bzip2, Snappy, Etc.
Cloud Network TopologyData Egress ChargesAvailability ZonesRegionsGCP-Specific Networking and Multiregional RedundancyDirect Network Connections to the CloudsCDNsThe Future of Data Egress Fees

Content preview from Fundamentals of Data Engineering

About the Authors

Joe Reis is a business-minded data nerd who’s worked in the data industry for 20 years, with responsibilities ranging from statistical modeling, forecasting, machine learning, data engineering, data architecture, and almost everything else in between. Joe is the CEO and cofounder of Ternary Data, a data engineering and architecture consulting firm based in Salt Lake City, Utah. In addition, he volunteers with several technology groups and teaches at the University of Utah. In his spare time, Joe likes to rock climb, produce electronic music, and take his kids on crazy adventures.

Matt Housley is a data engineering consultant and cloud specialist. After some early programming experience with Logo, Basic, and 6502 assembly, he completed a PhD in mathematics at the University of Utah. Matt then began working in data science, eventually specializing in cloud-based data engineering. He cofounded Ternary Data with Joe Reis, where he leverages his teaching experience to train future data engineers and advise teams on robust data architecture. Matt and Joe also pontificate on all things data on The Monday Morning Data Chat.