Data Governance with Unity Catalog on Databricks

Chapter 1. The Modern Governance Stack

Data is the new oil.

Clive Humby

In this chapter, we’ll start with an introduction to data governance and why it’s important. If you’re already familiar with data governance in general and convinced that it is truly important in the analytics space, you may skip to “The Dawn of the Lakehouse”. If you’re already familiar with the lakehouse paradigm and the Databricks Platform, then jump right into “Databricks Unity Catalog: Enabling Unified Governance”.

Introducing Data Governance

On October 28, 2021, a cyber security team at SafetyDetectives discovered an unsecured Amazon S3 bucket that contained over a million files. Among other things, the content of the bucket included personally identifiable information (PII) of employees and sensitive company data of some airports in Colombia and Peru. Although the contents of the buckets dated back to November 2018, no one knew how long the buckets had been publicly exposed. Apparently, the S3 bucket belonged to a well-established and prominent security services company called Securitas, headquartered in Stockholm, Sweden, with branches around the world.

This was not a standalone incident. In July 2019, Capital One suffered a data breach where over 100 million customer records were accessed due to a cloud firewall configuration vulnerability. Yet another incident related to Amazon S3 bucket misconfiguration led to the exposure of a significant amount of sensitive data in March 2022. The data exposed ...