RAG-Ready Patterns for Data Platforms

Preface

There are moments in technology when something new arrives and forces us to reconsider not just what we are building, but the assumptions underneath everything we have already built.

Generative AI has been one of those moments for us.

For years, we worked on large-scale data platforms with a fairly clear mission: bring data together, make it reliable, govern it, and make it useful. We built pipelines, semantic models, quality systems, catalogs, dashboards, and services. We learned how difficult it can be to create a common understanding of something as seemingly simple as a customer, a user, a product, or a subscription across a large organization.

Then, large language models (LLMs) changed the consumer of that data.

Suddenly, the question was no longer only whether a person could find the right dashboard or whether an analyst could write the right query. An AI system needed to retrieve information from across the enterprise, understand what that information meant, determine whether it was current and trustworthy, respect the conditions under which it could be used, and assemble all of that context quickly enough to answer a question.

That is an exciting change. It is also a profound architectural one.

We have seen what becomes possible when an AI system can reason over trusted enterprise context. Questions that once required navigating dashboards, finding datasets, consulting documentation, and tracking down experts can increasingly begin with natural language. Data that ...