book

Fundamentals of Data Observability

by Andy Petrella

August 2023

Beginner to intermediate

264 pages

7h 15m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Overview of the BookWho Should Read This BookConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
Scaling Data TeamsChallenges of Scaling Data TeamsSegregated Roles and Responsibilities and Organizational ComplexityAnatomy of Data Issues and ConsequencesImpact of Data Issues on Data Team DynamicsScaling AI RoadblocksChallenges with Current Data Management PracticesEffects of Data Governance at ScaleData Observability to the RescueThe Areas of ObservabilityHow Data Teams Can Leverage Data Observability NowLow Latency Data Issues DetectionEfficient Data Issues TroubleshootingPreventing Data IssuesDecentralized Data Quality ManagementComplementing Existing Data Governance CapabilitiesThe Future and BeyondConclusion
Channels of Data Observability InformationLogsTracesMetricsObservations ModelPhysical SpaceServerUserStatic SpaceDynamic SpaceExpectationsRulesAutomatic Anomaly DetectionPrevent Garbage In, Garbage OutConclusion
Data ArchitectureWhere Does Data Observability Fit in a Data Architecture?Data Architecture with Data ObservabilityHow Data Observability Helps with Data Engineering UndercurrentsSecurityData ManagementSupport for Data Mesh’s Data as ProductsConclusion
At the SourceGenerating Data Observations at the SourceLow-Level API in PythonDescription of the Data PipelineDefinition of the Status of the Data PipelineData Observations for the Data PipelineGenerate Contextual Data ObservationsGenerate Data-Related ObservationsGenerate Lineage-Related Data ObservationsWrap-Up: The Data-Observable Data PipelineUsing Data Observations to Address Failures of the Data PipelineConclusion
Abstraction StrategiesEvent ListenersAspect-Oriented ProgrammingHigh-Level ApplicationsNo-Code ApplicationsLow-Code ApplicationsDifferences Among Monitoring AlternativesConclusion
Introducing ExpectationsShift-Left Data QualityCorner Cases DiscoveryLifting Service Level IndicatorsUsing Data ProfilersMaintaining ExpectationsOverarching PracticesFail Fast and Fail SafeSimplify Tests and Extend CI/CDConclusion

Ingestion StageIngestion Stage Data Observability RecipesAirbyte AgentTransformationTransformation Stage Data Observability RecipesApache Sparkdbt AgentServingRecipesBigQuery in PythonOrchestrated SQL with AirflowAnalyticsMachine Learning RecipesBusiness Intelligence RecipesConclusion
Data TranslucenceOpaque SystemsSaaSDon’t Touch It; It (Kinda) WorksInherited SystemsStrategies for Data TranslucenceStrategiesThe Data Observability ConnectorExample: Building a dbt Data Observability Connector (SaaS)Conclusion
Unification of ProcessingGenerative MilestonesTrustable Expanded CreativityConclusion