Data Contracts

Chapter 3. The Challenges of Scaling Data Infrastructure

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 3rd chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

In this chapter we’ll dive further into the challenges of the everyday workflows of data producers and consumers in building and maintaining complex software and data systems. Furthermore, we’ll elaborate on how data development is different from software development, and the implications of these differences on our ability to drive value with data in an organization. Being aware of these challenges will equip you to understand how data contracts fit within the workflows of developers, what problems it mitigates, and how you can help developers understand their role within a data contract architecture.

How Data Development Is Not Like Software Development

As mentioned in Chapter 1, data development best practices proceeded software development by decades. There existed previously a set of best practices, design philosophies, and management methodologies wholly separate from ...