Automating Data Transformations

Preface

Many have heard the term data velocity, popularized by Oscar Herencia as a part of his five V’s of data: volume, velocity, variety, veracity, and value. For the past two decades, the first four V’s have grown exponentially, but what about the most important—value? Data transformation exists to deliver value and drive tangible improvements in business outcomes. While many organizations have built data warehouses and lakes, growing the volume of their information, how many have seen their data’s value grow proportionally?

Today, integrating data into business operations is table stakes. Winning organizations will have the most performant and scalable architectures, prioritize the value of their outputs, and place the greatest emphasis on results. We’ve seen this pattern in other operational groups: engineering, sales, and marketing, to name a few. From workplace wikis to a slew of Slack apps, automation and tooling drive process improvement, which multiplies the value-add of passionate, curious employees—this is the premise of the modern data stack (MDS).

The MDS is the practitioner’s toolbelt. Under its umbrella are products for data ingestion, storage, transformation, analytics, and governance. These solutions enable data teams to be lean and efficient: only a few years ago most were built from scratch, limiting robust data analysis to organizations with a fleet of data engineers and architects. As a result, those who leverage the MDS are capable of delivering more value ...