Data Engineering Design Patterns

Appendix. Summary of Patterns

It’s unlikely that you’ll work with all the patterns at once. You’ll face different patterns at different times throughout your data engineering journey, so there’s no need to remember all of them by heart. Instead, you should be able to find them easily and adapt them to the problem you’re currently facing.

To make this task easier, you can find here a summary of the design patterns presented in the book with quick reminders of their main use cases and gotchas.

Data Ingestion Design Patterns

Pattern name	Use case	Gotchas
Full Loader	Load full dataset	Increasing data volume for growing datasets Data consistency during the load process
Incremental Loader	Load chunks of a dataset	Loading physically deleted rows Volume of loaded data during a backfilling
Change Data Capture	Load chunks of a dataset as they come	Setup complexity (database layer) Changes scope, overall or only after the setup Data remains at rest
Passthrough Replicator	Replicate a dataset without altering any information	Serialization side effects, like badly formatted dates Production resources isolation PII data Latency impact if automated via infrastructure Metadata to replicate if relevant
Transformation Replicator	Replicate a dataset with a custom transformation	Higher risk of misformatted attributes due to the schema-based transformation PII data definition up to date
Compactor	Optimize storage of the ingested files	Compaction frequency impact ...