Financial Data Engineering

Chapter 8. Data Storage Layer

In the previous chapter, you learned how the data ingestion layer works, including the mechanisms, technologies, and formats used to ingest data into a financial data infrastructure. Once ingested, data must be stored and persisted in a storage location for further processing and querying. This is where the data storage layer comes into the picture.

To help you understand how to build a robust storage layer, this chapter will provide you with the necessary fundamentals and concepts, along with illustrations of technologies and their applications in finance. First, you’ll learn how to approach the design of a data storage system (DSS) using appropriate criteria. Next, the concept of a data storage model (DSM) and its categorization criteria will be introduced. Then, I will present a comprehensive list of DMSs relevant to the financial industry, highlighting each DMS’s key features, data modeling concepts, technical implementations, and financial applications.

Principles of Data Storage System Design

Throughout this book, I use the term data storage system (DSS) to denote a software implementation that enables the storage and retrieval of data. In many cases, people use the term “database” to refer to a storage solution. However, databases are only one type of DSS, albeit a popular one.

As a financial data engineer, knowing how to assess, choose, design, and implement a DSS should be one of your primary skills and areas of knowledge. The DSS is a ...