Automating Data Transformations

Chapter 1. Today’s Modern Data Stack

Quite often, data teams develop a myopic focus on the pursuit of the “perfect” process. While optimization is important, it’s easy to overlook why companies have sunk millions into data operations. Drawn to the attractiveness of real-time data or the hype of machine learning (ML), AI, and cutting-edge techniques, many seek overly complex solutions when value can be derived from much simpler processes.

The goal of the modern data stack (MDS) is to simplify and democratize access to insight that can enable any organization to improve decision making, delivering value to the business. Working backward to achieve our desired outcome, the tools that support a product-led data team must:

• Be simple to implement and easy to understand (democratize access to data)

• Scale with the growth of the company, both in head count and data maturity

• Limit technical debt and vendor lock-in

In this chapter, we’ll provide a brief overview of the MDS and walk through basic concepts before diving into the creation of a successful data framework. Afterward, we’ll discuss the importance of automation and what data transformation currently lacks.

What Is the MDS?

The MDS is a term pioneered by Fivetran to describe the solutions that comprise an organization’s system for capturing, enriching, and sharing data. We’ve seen tremendous advancement in data processes—only a few years ago, every element of data ingestion and transformation had to be built from scratch ...