Fundamentals of Data Observability

Chapter 8. Making Opaque Systems Translucent

In previous chapters, when I covered low-code no-code solutions, I enumerated some outside-of-the-box strategies that can be used to generate data observations until those solutions evolve to become natively data observable.

Other, often familiar, systems are similar to those solutions, as they have the same characteristics: the data usages happen in the engine under the hood. It is neither efficient (cost and time effective) nor recommended to open it.

Significant risks are associated with systems that do not provide clear visibility into their engine or usage of data. The main problem is that users cannot access the system to analyze if there are any issues that may cause problems. This lack of transparency means that users are unable to identify the root causes of issues, which forces them to patch downstream systems instead of addressing the upstream problems. Another issue with these systems is the loss of knowledge about their internal behavior.

While people may have configured or developed them successfully in the past, very few still have the internal knowledge to understand what they do. As a result, the strategies covered in Chapter 4 and Chapter 5 for achieving data observability may not apply. However, this chapter introduces strategies for these opaque systems, which allow users to get closer to data observability without necessarily satisfying all its principles.

These strategies will make the systems partially data observable ...