book

Driving Data Quality with Data Contracts

by Andrew Jones

June 2023

Intermediate to advanced

206 pages

5h 33m

English

Packt Publishing

Read now

Unlock full access

ForewordContributorsAbout the authorAbout the reviewers
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesConventions usedGet in touchShare Your ThoughtsDownload a free PDF copy of this book
The enterprise data warehouseThe big data platformThe modern data stackThe state of today’s data platformsThe lack of expectationsThe lack of reliabilityThe lack of autonomyThe ever-increasing use of data in business-critical applicationsSummaryFurther reading
What is a data contract?An agreed interface between the generators of data, and its consumersSetting expectations around that dataDefining how the data should be governedFacilitating the explicit generation of quality dataThe four principles of data contractsWhen to use data contractsData contracts and the data meshDomain ownershipData as a productSelf-serve data platformFederated computational governanceData contracts enable a data meshSummaryFurther reading
Using data contracts to change an organizationArticulating the value of your dataBuilding data productsWhat is a data product?Adopting a data product mindsetDesigning a data productWalking through an example of a data productSummaryFurther reading
Who is a consumer, and who is a generator?Data consumersData generatorsAssigning responsibility and accountabilityFeeding data back to the product teamsManaging the evolution of dataSummaryFurther reading
Why we need data governanceThe requirements of data governanceHow data governance programs are typically appliedPromoting data governance through data contractsAssigning responsibility for data governanceResponsibilities of the data generatorsIntroducing the data architecture councilWorking together to implement federated data governanceSummaryFurther reading

The schema of a data contractDefining a schemaUsing a schema registry as the source of truthEvolving your data over timeEvolving your schemasMigrating your consumersDefining the governance and controlsSummaryFurther reading
A step-change in building data platformsBuilding generic data toolingIntroducing a data infrastructure teamA case study from GoCardless in promoting autonomyPromoting autonomy through decentralizationIntroducing the principles of a contract-driven data architectureAutomationGuidelines and guardrailsConsistencyProviding self-served data infrastructureSummaryFurther reading
Technical requirementsCreating a data contractProviding the interfaces to the dataIntroducing IaCCreating the interfaces from the data contractCreating libraries for data generatorsPopulating a central schema registryRegistering a schema with the Confluent schema registryManaging schema evolutionImplementing contract-driven toolingSummaryFurther reading
Getting started with data contractsThe ability to define a data contractThe ability to provision an interface for the data for consumers to queryThe ability of generators to write data to the interfaceMigrating to data contractsDiscovering data contractsWhat is a data catalog?Why are data catalogs important for discovering data contracts?What is data lineage?Why is data lineage important for data contracts?Building a mature data contracts-backed data cultureSummaryFurther reading
Designing a data contractIdentifying the purposeConsidering the trade-offsDefining the data contractDeploying the data contractMonitoring and enforcing data contractsThe data contract’s definitionThe quality of the dataThe performance and dependability of the dataData contract publishing patternsWriting directly to the interfaceMaterialized views on CDCThe transactional outbox patternThe listen-to-yourself patternSummaryFurther reading
Other Books You May EnjoyPackt is searching for authors like youShare Your ThoughtsDownload a free PDF copy of this book

Overview

Modern organizations often face challenges with data quality and reliability. 'Driving Data Quality with Data Contracts' offers a comprehensive approach to overcoming these hurdles by introducing data contracts-a new paradigm aiming to enhance data accountability and trustworthiness. You'll learn practical methods for implementing effective data platforms built on data contracts.

What this Book will help me do

Understand the challenges of contemporary data architectures and how data contracts address these issues.
Learn the components of a data contract and the benefits it brings to both data generators and consumers.
Gain practical knowledge and see working implementations of data contracts in real-world scenarios.
Explore strategies for embedding data governance into your organization's culture through data contracts.
Design and implement a cutting-edge data platform leveraging the advantages of data contracts.

Author(s)

Andrew Jones brings years of experience in data engineering and architecture. As a thought leader in data practices, he has spearheaded data contracts' implementation to enhance organizational data reliability and trust. Andrew's writing combines technical depth with practical insights to guide readers through solutions to their data challenges.

Who is it for?

This book is ideal for data engineers, architects, and practitioners seeking to enhance their organization's data platforms. Readers with experience in data management or engineering will gain actionable insights into improving data quality. It's also relevant for product managers and executives wanting to promote efficient and accountable data culture. Beginners in modern data practices will find it accessible yet insightful.