97 Things Every Data Engineer Should Know

Chapter 19. Data Pipeline Design Patterns for Reusability and Extensibility

Mukul Sood

Designing extensible, modular, reusable data pipelines is a large topic that’s relevant in data engineering, as it requires dealing with constant change across different layers such as data sources, ingestion, validation, processing, security, logging, and monitoring. These changes happen at varying rates across the layers and impact data pipelines differently depending on the pipeline’s level of abstraction and design.

To provide context for the layers of a data pipeline and to start to map the configuration, the pipeline can be seen in a distilled form as comprising Ingestion, Processing, and Result layers. For each layer, we can think in terms of functions that map to functional blocks. The content of the blocks changes depending on the layer requirements. This helps us think in terms of templates and configuration that could represent the pipeline’s directed acyclic graph (DAG).

The Ingestion, Processing, and Result layers could be mapped to different loggers and monitors based on requirements. For example, in the Ingestion layer, the file log could be S3, the event log could be custom, and the monitors could be Google Cloud operations and Redash. However, the Results layer could be mapped to a DataDog event log and event monitor.

If we take a generic pipeline, specifically looking at logging ...