Data Contracts

Chapter 1. Why the Industry Now Needs Data Contracts

Unfortunately, data quality and its foundations, such as data modeling, have been severely deprioritized with the rise of big data, cloud computing, and the Modern Data Stack. Though these advancements enabled the prolific use of data within organizations and codified professions such as data science and data engineering, data’s ease of use also came with a lack of constraints, leading many organizations to take on substantial data debt. With pressure for data teams to move from doing R&D to actually driving revenue, as well as the shift from model-centric to data-centric AI, organizations are once again accepting the merits of data quality being a must-have instead of a nice-to-have. Driving this even further is the tight coupling between data produced or ingested upstream, typically by software engineers, and downstream data teams that have historically been siloed from each other.

While these all sound like explicitly technical problems, at their root they come down to issues with change management within complex systems. Thus, we believe that data contracts—agreements between data producers and consumers that are established, updated, and enforced via an API—are necessary for scaling and maintaining data-related change management within an organization. Before going in depth about what data contracts are and how to implement them, this chapter highlights why our industry has forgone data quality best practices, why we’re ...