Automating Data Transformations

Chapter 4. Summary and Further Reading

In our report, we sought to tie transformation to the act of creating value from data. Underlying this daunting task is a set of prerequisite organizational characteristics: thinking about “data as a product” (a core tenet of data mesh); building a data-driven culture; enabling DAaaS; and building data from a column-aware, metadata-first framework.

While a number of guiding principles and prevailing philosophies help the data practitioner in their journey, every situation is unique, demanding a bespoke solution. It’s our hope that this report can help your organization achieve efficiency and automation in transformation, unlocking the full value of the modern data stack and providing a lens into the past and a vision for the future of data transformation.

To further your understanding, we highly recommend the following as further reading:

Data Mesh by Zhamak Dehghani (O’Reilly, 2022)

This book was revolutionary in its introduction to the DaaP concept, and Zhamak Dehghani’s approach to a decentralized data team of the future is applicable to most data teams. For those looking to build out their organization or restructure, Data Mesh is a must-read.

Data Pipelines Pocket Reference by James Densmore (O’Reilly, 2021)

A down-to-earth manual on how to solve common data problems in the pipeline step, this text walks through pipelines—from definition to implementation—with considerations for maintenance, testing, and alerting. This text ...