Automating Data Transformations

Chapter 3. Delivering Value with Data Transformations Through Automation

Though there are numerous ways for data systems to create value, each follows a core set of principles. A successful operation needs to be built on simplicity, flexibility, user-friendliness, and a metadata-first approach. A foundation in metadata is essential. Specifically, metadata originating at the column level allows for the precise dissemination of business context. While wikis and ad hoc questions might work for startups, this quickly becomes untenable at large enterprises.

More importantly, a metadata-based implementation enables a data architecture as a service (DAaaS) approach. As we’ll present in the following section, DAaaS leverages “data patterns” that can be used to break down data silos, eliminate analytics bottlenecks, and automate the many pain points that exist in the transformation layer today.

Principles of Data Value

The concept of providing value with data is not new, though it has grown in popularity and depth in the last few years. We feel data value is best approached through the data mesh framework. When viewed through this lens, it becomes apparent that a decentralized approach will be transformative in the data space. Decentralization of data skill, combined with a column-aware architecture and automation in the transformation layer, will serve to deliver value in the most efficient way possible.

Product-First

In Chapter 1 we introduced the concept of data as a product (DaaP), ...