Financial Data Engineering

Chapter 7. Data Ingestion Layer

In Chapter 2, we learned about the different sources and mechanisms that generate financial data. This included public, market, alternative, and internal sources. Once data is generated at the source, its lifecycle within a financial data infrastructure begins with ingestion. As this chapter will show, data ingestion isn’t as simple as the term may sound. In today’s complex financial data landscape, data ingestion has expanded to encompass a large variety of data transmission and arrival processes as well as ingestion technologies, mechanisms, and formats.

Data ingestion serves as the foundational layer for information exchange in financial market operations. It facilitates communication among financial institutions and market participants for initiating transactions like payments, settlements, and trades. It also supports the exchange of inquiries and notifications between financial entities. Furthermore, compliance with regulatory requirements relies on efficient data ingestion practices, enabling financial firms and brokers to transmit various financial details and compliance reports to regulatory bodies. In addition, data ingestion is essential for disseminating market data and delivering real-time updates, critical for maintaining liquidity and operational efficiency throughout financial markets.

This chapter will examine the data ingestion layer as the primary entry point for receiving transmitted data and as a crucial bridge facilitating ...