Data Governance with Unity Catalog on Databricks

Chapter 7. Observability and Discoverability

People spend 60% to 80% of their time trying to find data. It’s a huge productivity loss.

Dan Vesset, Group Vice President, IDC

Before joining Databricks, Kiran worked on an on-premises data platform running Apache Spark for almost four years. One of the key issues he faced while running the platform, supporting multiple users who did regulatory reporting on the data, was data quality. Working on near real-time data processing use cases, particularly those that rely on message processing systems such as Apache Kafka, Azure Event Hubs, or AWS Kinesis, further aggravated the problem. The risk of missing a message in transit from the message processing queue without anyone knowing its absence was exceptionally high. Almost 95% of the time, the issues were attributed to low data discoverability in the data platform. The support team developed, deployed, and monitored custom tools to prevent data loss issues. The team used multiple monitoring tools for different platform aspects, which complicated the overall process. Building and deploying these tools helped curb data issues but came with the overhead of managing the infrastructure and additional costs.

When building a business application using data, it’s essential to prioritize data quality from the outset, rather than treat it as an afterthought. Data quality is just one part of the equation. Once you have solved it, the next critical issue concerns data discoverability and platform ...