Data Contracts

Preface

If you have picked up this book, it’s likely that you’ve deeply felt the pain of managing data while lacking control of your data’s ingestion and generation. Though at one point our industry centered around well-thought-out on-prem implementations with robust data models, the rise of cloud computing and the explosion of data products within organizations, via AI, have incentivized speed to market at the cost of turning the data layer into chaos. Many data teams within this situation find themselves constantly being reactive, repeatedly fixing the next data-related fire within the company.

At its core, we believe this challenge in our industry stems from the difficulty of change management between historically siloed teams of data producers and consumers. Specifically, there is a disconnect between upstream application code, which defines how data is captured within a software system, and the downstream data products that leverage this data. We argue that data contracts serve as a mechanism for aligning data producers and consumers through automation and defining expectations as code.

What Are Data Contracts?

Data contracts are an architecture pattern that enables an agreement between data producers and consumers that is established, updated, and enforced via an API. They’re part of a larger movement called shift left, where you use automation to enable upstream software developers to account for required enforcement pertinent to their domain—this approach was first validated ...