Fundamentals of Data Observability

Chapter 6. Implementing Expectations

Like Chapter 2, we end this section on implementation with expectations. As data practitioners, expectations are key to the success of our data initiatives, especially their operational maintenance and keeping our stakeholders’ trust. Introducing them in our applications is an important step in their development and maintenance, just as important as tests and documentation.

This chapter highlights moments of the application lifecycle where we must introduce expectations and the associated practices to discover and implement them. Expectations also represent pieces of knowledge or interpretations of the constraints based on understanding the context. They have their maintenance process just as tests and documentation do.

To conclude the chapter, we present two examples of overarching practices that combine the benefits of generating observations at the source and introducing expectations. But first, let’s see how to define expectations and add them to our data applications.

Introducing Expectations

As mentioned in Chapter 2, expectations represent data constraints or usage that we must respect. If we don’t follow them, we must be aware and take action. This section explains how to turn the expectations into rules—how to define and implement them—and the actions we need to take.

Before jumping in, let’s see what a rule looks like. You can implement a rule as a function that takes observations as inputs and computes a Boolean condition in its ...