book

Data Contracts

by Chad Sanderson, Mark Freeman, B. E. Schmidt

November 2025

Intermediate to advanced

348 pages

10h 44m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Foreword
Preface
What Are Data Contracts?How to Use This BookPart I: Introduction to the Data Contract ArchitecturePart II: Implementation of the Data Contract ArchitecturePart III: Getting Leadership Buy-in for the Data Contract ArchitectureConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
I. Introduction to the Data Contract Architecture
1. Why the Industry Now Needs Data Contracts
Garbage-In, Garbage-Out CycleModern Data ManagementWhat Is Data Debt?How Garbage-In, Garbage-Out CompoundsThe Death of Data WarehousesThe Pre-Modern EraSoftware Eats the WorldA Move Toward MicroservicesData Architecture in DisrepairRise of the Modern Data StackThe Big PlayersRapid GrowthProblems in ParadiseData-Centric AI and the Rise of Shift Left Data PracticesDiminishing ROI of Improving ML ModelsCommoditization of Data Science WorkflowsData’s Rise Over ML in Creating a Competitive AdvantageThe Rise of Shift Left Data PracticesConclusion
2. Data Quality Isn’t About Pristine Data
Defining Data QualityOLTP Versus OLAP and Its Implications for Data QualityA Brief Summary of OLTP and OLAPTranslation Issues Between OLTP and OLAP Data WorldviewsThe Cost of Poor Data QualityMeasuring Data QualityWho Is ImpactedConclusion
3. The Challenges of Scaling Data Infrastructure
How Data Development Is Not Like Software DevelopmentHow Software Engineers Build ProductsHow Data Developers Build ProductsCore Challenges for Modern Data Engineering TeamsWhy Data Development Needs a Design SurfaceThe Cost of Large-Scale RefactorsLarge-Scale Refactor ConsiderationsUse Case: Alan’s Large-Scale RefactorThe Dangers of Database MigrationsThe Role of Change Management in Data QualityThe Entropic Behavior of DataHow Data Drifts from Established Business LogicChange Management Needs to Align with the Needs of the BusinessHow Infrastructure Needs Change at ScaleDunbar’s Number and Conway’s LawCase Study: Atlassian Engineering TeamHow Data Contracts Enable Change Management at ScaleConclusion
4. An Introduction to Data Contracts
Collaboration Is Different in DataThe Stakeholders You Will Work WithThe Role of Data ProducersThe Role of Data ConsumersThe Impact of Producers and ConsumersThe Trials and Tribulations of Data Consumers Managing Data QualityAn Alternative: The Data Contract WorkflowSteps of the Data Contract WorkflowWhere to Implement Data ContractsThe Maturity Curve of Data Contracts: Awareness, Ownership, and GovernanceOutcomes of Implementing Data ContractsData Contracts Versus Data ObservabilityConclusion
II. Implementation of the Data Contract Architecture
5. The Data Contract Components: Data Assets and Contract Definition
Overview of ComponentsData AssetsAnalytics DatabasesTransactional DatabasesEvent Sourcing and Event StreamsFirst-Party Data, Third-Party PlatformContract DefinitionData Contract SpecBusiness LogicSchema Registry and Data CatalogsConclusion
6. The Data Contract Components: Detection and Prevention
DetectionData QuarantiningChange Data Capture, Stream Processing, and Live Data MonitoringEnd-to-End Lineage: Source Code Flow Graphs and Data LineageStatic Code AnalysisPreventionVersion ControlCI/CDViolation Monitoring and AlertsConclusion

7. Implementing Data Contracts
Mock Data Contract Scenario IntroDatasetMuseum ApplicationProject Repo OverviewImplementing Data ContractsData Contracts Architecture OverviewComponent A: Data AssetsComponent B: Contract DefinitionComponent C: DetectionComponent D: PreventionPutting It All TogetherData Contract Violation ScenarioReceived Request to Normalize the object_images TableUpdate Data Migration File for New SchemaCheck Unit Tests and Trigger Contract ViolationPush to Remote Branch and View Error LogsRevert Database Migration ChangesDiscuss with Downstream Team to Learn Use CaseCreate New Data Contract for the object_images_normalized TableRun Unit Tests and See Error for Missing Asset in CatalogFix Database Migration Until All Checks PassPush to Remote Branch and See It PassConclusion
8. Real-World Case Studies of Data Contracts in Production
Convoy Data Contract StoryGlassdoor Data Contract StoryAdevinta Spain Data Contract StoryConclusion
III. Getting Leadership Buy-in for the Data Contract Architecture
9. Shift Left: The Cultural Change Needed for Data Contracts
What Needs to Be True: What We Missed About Shift Left DataUnderstanding the Stakes: Three Real-World Examples of Perception Gap ComplicationsPerception Gaps Fuel Faulty AssumptionsPerception Gaps Obscure Real-World ComplexityPerception Gaps Can Pit Teams Against Each OtherEnabling Adoption: A Strategic Approach for Closing the Software-Data GapFinding a Bridge and Fostering TrustCharting the Course with Objective InformationEvolving Shift Left Data Alongside Your OrganizationMeasuring SWE Quality-of-Life Improvements During Shift Left Data AdoptionWhat the Rising Need for DevSecOps DemonstratedSeeing (and Embracing) Shift Left Data for What It Truly IsConclusion
10. Change Management: The Crux of People, Process, and Technology
The Importance of Change ManagementData as Supply ChainsLevels of Data Contract ImplementationsAirplane and Airline ProjectsForward- and Backward-Looking ProblemsChallenges of Technology-First ImplementationsThe Trap of StandardsRequirements of a Successful Data Contract ImplementationDeveloping a Strategy for Introducing Data ContractsDefine the ProblemStructure the ProblemPrioritize the IssuesDevelop an Issue Analysis Work Plan and Conduct the AnalysesSynthesize Your FindingsDevelop RecommendationsConclusion
11. Creating Your First Wins with Data Contracts
Determining Your First Data Contract Use CaseDefining Data ProductsIdentifying Your Steel ThreadsMaking Your Final DecisionDetermining Your Business Case and RequirementsGather Information and Plan to EngageMap Out Where Key Stakeholders Fit in the Data LifecycleDetermine Key Stakeholder Influence, Interest, and RisksGain Executive Leadership SponsorshipEight Steps Toward Organization-Wide Shift Left AdoptionEstablishing a Sense of UrgencyCreating the Guiding CoalitionDeveloping a Vision and StrategyCommunicating the Change VisionEmpowering Employees for Broad-Based ActionGenerating Short-Term WinsConsolidating Gains and Producing More ChangeAnchoring New Approaches in the CultureAttribution in Action: The Importance of Making Success TangibleConclusion
12. Measuring the Impact of Data Contracts
What’s Worth Measuring?Where to Start If You Have Minimal Existing MeasurementsMinimal Viable Metric TreesMeasuring Across the Three Phases of AdoptionPhase 1 Goal: Measuring for ViabilityPhase 2 Goal: Measuring for RepeatabilityPhase 3 Goal: Measuring for ScalabilityStorytelling for Measurement ImpactDepartureTransformationReturnPutting It All TogetherConclusion
Conclusion: AI and the Future of Data Contracts
Future Implications of Shift Left DataData as CodeThe Impact of AIThe Role of AgentsLLMs and ContextThe Future of Data ContractsNext Steps
Appendix. Additional Resources
Chapter 1Chapter 2Chapter 3Chapter 5Chapter 9Chapter 11Chapter 12
Index
About the Authors

Content preview from Data Contracts

Appendix. Additional Resources

This book has an accompanying website that provides additional articles from the authors, as well as corresponding videos to guide your reading throughout the chapters. There is also a GitHub repository with a sandbox environment you can run locally or within a browser to see how to implement data contracts and the data contract violation workflow.

In additon, we recommend the further readings listed here by chapter.

Chapter 1

Chapter 2

Chapter 3

James Ivers et al., “Industry Experiences with Large-Scale Refactoring” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2022), Association for Computing Machinery (2022): 1544–1554.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Driving Data Quality with Data Contracts

Publisher Resources

ISBN: 9781098157623Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business