book

Automating Data Quality Monitoring

by Jeremy Stanley, Paige Schwartz

January 2024

Intermediate to advanced

220 pages

6h 3m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Who Should Use This BookConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
High-Quality Data Is the New GoldData-Driven Companies Are Today’s DisruptersData Analytics Is DemocratizedAI and Machine Learning Are DifferentiatorsCompanies Are Investing in a Modern Data StackMore Data, More ProblemsIssues Inside the Data FactoryData MigrationsThird-Party Data SourcesCompany Growth and ChangeExogenous FactorsWhy We Need Data Quality MonitoringData ScarsData ShocksAutomating Data Quality Monitoring: The New Frontier
Monitoring RequirementsData Observability: Necessary, but Not SufficientTraditional Approaches to Data QualityManual Data Quality DetectionRule-Based TestingMetrics MonitoringAutomating Data Quality Monitoring with Unsupervised Machine LearningWhat Is Unsupervised Machine Learning?An Analogy: Lane Departure WarningsThe Limits of AutomationA Four-Pillar Approach to Data Quality Monitoring
Assessing Your DataVolumeVarietyVelocityVeracitySpecial CasesAssessing Your IndustryRegulatory PressureAI/ML RisksData as a ProductAssessing Your Data MaturityAssessing Benefits to StakeholdersEngineersData LeadershipScientistsConsumersConducting an ROI AnalysisQuantitative MeasuresQualitative MeasuresConclusion
RequirementsSensitivitySpecificityTransparencyScalabilityNonrequirementsData Quality Monitoring Is Not Outlier DetectionML Approach and AlgorithmData SamplingFeature EncodingModel DevelopmentModel ExplainabilityPutting It Together with PseudocodeOther ApplicationsConclusion
Data Challenges and MitigationsSeasonalityTime-Based FeaturesChaotic TablesUpdated-in-Place TablesColumn CorrelationsModel TestingInjecting Synthetic IssuesBenchmarkingImproving the ModelConclusion
How Notifications Facilitate Data Issue ResponseTriageRoutingResolutionDocumentationTaking Action Without NotificationsAnatomy of a NotificationVisualizationActionsText DescriptionWho Created/Last Edited the CheckDelivering NotificationsNotification AudienceNotification ChannelsNotification TimingAvoiding Alert FatigueScheduling Checks in the Right OrderClustering Alerts Using Machine LearningSuppressing NotificationsAutomating the Root Cause AnalysisConclusion
Monitoring Your Data StackData WarehousesIntegrating with Data WarehousesSecurityReconciling Data Across Multiple WarehousesData OrchestratorsIntegrating with OrchestratorsData CatalogsIntegrating with CatalogsData ConsumersAnalytics and BI ToolsMLOpsConclusion
Build Versus BuyVendor Deployment ModelsConfigurationDetermining Which Tables Are Most ImportantDeciding What Data in a Table to MonitorConfiguration at ScaleEnablementUser Roles and PermissionsOnboarding, Training, and SupportImproving Data Quality Over TimeInitiativesMetricsFrom Chaos to Clarity

Table IssuesLate ArrivalSchema ChangesUntraceable ChangesRow IssuesIncomplete RowsDuplicate RowsTemporal InconsistencyValue IssuesMissing ValuesIncorrect ValuesInvalid ValuesMulti IssuesRelational FailuresInconsistent Sources

Content preview from Automating Data Quality Monitoring

About the Authors

Jeremy Stanley is cofounder and CTO at Anomalo. Prior to Anomalo, Jeremy was the VP of Data Science at Instacart, where he led machine learning and drove multiple initiatives to improve the company’s profitability. Previously, he led data science and engineering at other hypergrowth companies like Sailthru and Collective. He has applied machine learning and AI technologies to everything from insurance and accounting to ad tech and last-mile delivery logistics. He’s also a recognized thought leader in the data science community with hugely popular blog posts like “Deep Learning with Emojis (Not Math)”. Jeremy holds a BS in mathematics from Wichita State University and an MBA from Columbia University. You can find Jeremy on LinkedIn and on X (formerly known as Twitter).

Paige Schwartz is a professional technical writer at Anomalo who has written for clients such as Airbnb, Grammarly, and OpenAI. She specializes in communicating complex software engineering topics to a general audience and has spent her career working with machine learning and data systems, including five years as a product manager on Google Search. She holds a joint BA in computer science and English from UC Berkeley.