Unlock Complex and Streaming Data with Declarative Data Pipelines

Introduction

“Data is the new oil.” Companies must be “data-driven or die.” While these sayings have become clichés, they do express a truth. Our physical and digital worlds are increasingly recorded as data, and there is substantial value in converting that data into information about our business or external factors impacting our business.

However, making modern data—that is, complex and streaming data—useful has become more challenging due to rapid increases in data volume, velocity, and variety that began a decade ago. In the old days (20 years ago), the only data you could access was stored in an on-premises relational database, which was a tightly coupled storage and compute environment loaded with curated, structured data. That data could be both prepared for analysis and queried by using SQL and batch processing.

At that time, the concept of a data pipeline didn’t exist. You had “ETL jobs” that moved data from various sources into a database or data warehouse while performing some light processing—cleansing, normalization, field mapping—on the data.

Today we have uncurated data, emitted in real time by digital systems and devices and stored in its raw format in a data lake. This has created amazing opportunities for analytics and technology innovation, but it also has led to skills challenges for building and managing the continuous data pipelines required to process this data into an analytics-ready state.

Unlike ETL jobs, a modern data pipeline performs continuous, ...