Data Management at Scale

Chapter 10. Democratizing Data with Metadata

In this chapter we will look more closely at metadata’s role in the architecture and at how it must be managed. Metadata, as you have learned, describes all relevant aspects of the new architecture. It binds everything together and is key for delivering the insight, control, and efficiency large enterprises are looking for.

Metadata, on the other hand, is also a complicated subject to manage, scattered as it is across many tools, applications, platforms, and environments. Typically, a multitude of organized metadata repositories coexist in a large data architecture. Most metadata is also tightly coupled to a specific vendor product. Its volume and diversity are huge, so metadata usually needs to be properly selected, organized, and integrated before it can be managed. In this chapter you will learn what to focus on and learn the core ingredients of a good enterprise metadata model.

You’ll need to automate the collection, discovery, maintenance, and use of metadata to make it an integral part of your architecture. So we’ll also look at integration patterns and discuss why certain metadata must be persisted centrally. (To avoid reinventing the wheel, it’s best to reuse as many integration patterns as possible.)

Last but not least, we will look at data democratization, knowledge graphs, and how metadata can be used to drive data governance. To facilitate this, metadata has to be ubiquitous on an enterprise level and open and accessible ...