Fundamentals of Data Observability

Chapter 3. Roles of Data Observability in a Data Organization

In the previous chapter, you learned the definition of data observability and how data technologies and teams can embrace it. In this chapter, I will systematically look at data observability to analyze how it fits into the data organization, such as its data architecture and culture. Because the culture of data is a complex system itself, I will address it in two parts:

• Through the lens of data engineering roles in a data organization.

• The role of data in the evolution of our economy (and, thus, organizations).

Data Architecture

Data architecture is a critical component of a data organization. It sets the foundation of how data will deliver value for each department individually and collaboratively. The intent of this section is not to define the ideal data architecture, if there is such a thing, but to review where data observability must be introduced to support the sustainable management of data usage in the organization.

Not all organizations are equal regarding data architecture; some already have an established architecture expected to evolve or be replaced partially or entirely, and others are simply starting from a white page. Therefore, I will explain why you must consider data observability from the start when building a new architecture and how, despite its central role, it can be introduced as an add-on to existing data architecture.

Where Does Data Observability Fit in a Data Architecture?

To address ...