Accelerating Data Pipeline Development

Chapter 2. Coalesce Core Concepts

In the previous chapter, we laid the foundation for your understanding of Coalesce components, such as projects, workspaces, and storage locations. In this chapter, we will set the building blocks on top of your foundation by providing you with the core concepts that empower Coalesce data development. You will learn why column-aware architecture is important within data transformations, what nodes are and how they harness column-aware architecture, the data pipeline approach, and managing data development.

At the end of the chapter, you’ll walk away with the knowledge that will provide you with the framework you need to understand and develop your data on the platform. And there is no better place to start than understanding column-aware architecture.

Column-Aware Architecture

Data processing workloads in modern data platforms don’t just operate on the scale of thousands of database tables; they run on hundreds of thousands—even millions—of columns. Coalesce is built from the ground up to automate data transformations while supporting both a code-first approach and an intuitive visual interface. This all starts with a column-aware architecture.

But what is column-aware architecture? Column-aware architecture, or being column-aware, is an approach to managing data transformation with an understanding of columns and how they are connected. With this understanding, Coalesce provides automated column-level lineage while enabling the creation and ...