Data Management at Scale, 2nd Edition

Chapter 9. Democratizing Data with Metadata

Many of us have witnessed the power of OpenAI. In the future, metadata and artificial intelligence will intersect. What do you think this powerful AI will be capable of when it starts learning more about metadata? The impact could be immense. It could change the way we currently do data integration; AI models will be able to interpret language and predict and write code. AI might be able to fix data quality issues or give recommendations on how best to organize and structure your data landscape based on the lineage. Chief data officers need to prepare for these trends. As they will learn, metadata will be at the heart of it all.

In this chapter, we will look more closely at metadata’s role in a modern data architecture and how it should be managed. Metadata, as you have learned, describes all the relevant aspects of the new architecture. It binds everything together and is key for delivering the insights, control, and efficiency large enterprises are looking for.

Metadata is complicated to manage, scattered as it is across many tools, applications, platforms, and environments. Typically, a multitude of organized metadata repositories coexist in a large data architecture. Most metadata is also tightly coupled to a specific vendor product. Because of its great volume and diversity, metadata usually needs to be properly selected, organized, and integrated before it can be managed. Part of this chapter will be dedicated to the core ingredients ...