The Enterprise Data Catalog

Foreword

When I began focusing on data cataloging in the mid-2010s, the world of data analytics had reached an inflection point. The great modern data infrastructure projects, centered on data lakes and usually tied to internal Hadoop clusters, had reached a level of maturity. The technical architecture was in place. The promise of innovation and value creation, however, which had been the initial argument in favor of massive spending in the infrastructures of organizations, had hardly borne fruit beyond a few use cases driven by enthused pioneering teams.

I feel this relative failure has its roots more in the governance of these new transverse infrastructures than in the technology per se. These failures manifested themselves in two opposing ways.

In some cases, the lack of governance caused data lakes to morph into data swamps—enormous storage spaces containing data whose content and origin were unknown to everyone, and which nobody knew how to use.

In other cases, the deployment of sophisticated governance tools locked the data within complex and bureaucratic procedures, thus inhibiting the agility and experimentation that are needed at the heart of innovation.

As a result, organizations began migrating (or creating) their analytic infrastructures to the cloud (the state of the art of cloud solutions has massively improved since the middle of the 2010s) and rethinking how best to manage the huge volumes of data they needed to exploit.

There is, of course, plenty to say about ...