Financial Data Engineering

Chapter 1. Financial Data Engineering Clarified

Given all the payments, transfers, trades, and numerous financial activities that take place on a daily basis, can you imagine how much data the global financial sector generates? According to a 2011 report by McKinsey Global Institute, the banking and investment sector in the US alone stores and manages more than one exabyte of data. To put that in perspective, an exabyte is the equivalent of one billion gigabytes, and it translates into trillions of digital records. The same report shows that on average, financial services firms generate and store more data than firms in other sectors. Some statistics are even more astonishing; for instance, JPMorgan Chase, the largest bank in the United States by market capitalization, manages more than 450 petabytes of data. Bank of New York Mellon, a global financial services company specializing in investment management and investment services, manages over 110 million gigabytes of global financial data.

Naturally, we might extrapolate these estimates and figures to tens or even hundreds of exabytes if we take into account the global context and the constantly expanding financial landscape. As a result, data sits at the heart of the financial system, serving as both the input for different financial operations and the output generated from them. Importantly, to guarantee a healthy and well-functioning system, a reliable and secure data infrastructure is needed for generating, exchanging, storing, ...