Building Medallion Architectures

Chapter 11. Scaling the Medallion Architecture

In Parts I and II, we focused on constructing a single Medallion architecture. Yet, in the real world, organizations often manage multiple Medallion architectures to meet the diverse needs of various teams and data domains. This isn’t just speculation; it’s a fact well-known to anyone with hands-on enterprise experience. As organizations expand, they must scale their data management to support more data, more users, and more varied use cases. Thus, it’s common for enterprises to have decentralized architectures rather than a one-size-fits-all centralized solution.

However, there’s a debate out there. Some experts push for full decentralization across every aspect of technology, organization, and governance. But that’s not necessarily the best approach. Decentralization has its complexities, especially in managing processes and oversight. The key is to strike the right balance, as the degree of decentralization will differ between organizations. To delve deeper into this subject, we’ll explore the following two topics:

1. Scaling through decentralized data management using multiple Medallion architectures.

2. Scaling through Medallion inner architecture variations, which includes topics such as master data management (MDM) and enterprise data models.

By the end of this chapter, you will have a clear understanding of how to scale the Medallion architecture effectively and manage multiple Medallion architectures within the same organization. ...