Unlock Complex and Streaming Data with Declarative Data Pipelines

Conclusion

A Declaration of Independence for Data Engineers

Data engineering has become a very popular and technically challenging field. The amount of work is growing quickly due to the emergence of modern data and the demand for analytics that takes advantage of its ability to move beyond what happened—the purview of transactions—to why it happened—the place where analysis of the interactions that drive those transactions reigns.

New patterns that provide the foundation for reliable real-time analytics—such as event sourcing and stateful stream processing—are functions databases or data warehouses weren’t designed for. These add to the challenge.

Much of the manual effort required of data engineers to build modern data pipelines is a consequence of the incomplete evolution of big data systems that started with Apache Hadoop in 2006. In databases, these functions are automated “behind the scenes” through an “optimizer” function. When it comes to modern data systems, the data engineer is the “optimizer”!

The importance of declarative pipelines for big data and streaming data is that they mark a return to the model of automated optimization. This has two critical implications. First, it frees data engineers from the tedious effort of writing code, optimizing file systems, or orchestrating DAGs, allowing them to focus on more valuable data modeling and transformation design. Second, it allows anyone who knows SQL to build a modern data pipeline—including data scientists, analysts ...