Financial Data Engineering

Chapter 6. Overview of the Financial Data Engineering Lifecycle

As a financial data engineer, navigating the multitude, diversity, and complexity of available technological options can be overwhelming. Without a systematic approach in mind, this complexity may lead to chaotic situations and accumulating costly technical debt. Therefore, this chapter introduces a structured approach to financial data engineering, organizing its components into a layered architecture called the financial data engineering lifecycle (FDEL). This framework draws inspiration from the foundational work of Joe Reis and Matt Housley on data engineering lifecycles.

In this chapter, I will introduce the FDEL and outline its four layers: ingestion, storage, transformation and delivery, and monitoring. Following this, I will discuss specific criteria that financial data engineers can consider when selecting technologies to support the FDEL. Please note that since there is so much information to cover, I’ll save the details of each FDEL layer for Chapters 7 through 10.

Financial Data Engineering Lifecycle Defined

Data engineering is a fast-moving and continuously evolving field. However, if there is one constant that characterizes and defines the job of a data engineer, it is the fact that it revolves around systems that perform a series of actions for the extraction, transformation, storage, and consumption of data.

To move beyond the simple view of data engineering as merely a series of data-related tasks, ...