Data Warehousing with Greenplum, 2nd Edition

Foreword to the Second Edition

My journey with Pivotal began in 2014 at Morgan Stanley, where I am the global head of database engineering. We wanted to address two challenges:

• The ever-increasing volume and velocity of data that needed to be acquired, processed, and stored for long periods of time (more than seven years, in some cases)

• The need to satisfy the growing ad hoc query requirements of our business users

Nearly all the data in this problem space was structured, and our user base and business intelligence tool suite used the universal language of SQL. Upon analysis, we realized that we needed a new data store to resolve these issues.

A team of experienced technology professionals spanning multiple organizational levels evaluated the pain points of our current data store product suite in order to select the next-generation platform. The team’s charter was to identify the contenders, define a set of evaluation criteria, and perform an impartial evaluation. Some of the key requirements for this new data store were that the product could easily scale, provide dramatic query response time improvements, be ACID and ANSI compliant, leverage deep data compression, and support a software-only implementation model. We also needed a vendor that had real-world enterprise experience, understood the problem space, and could meet our current and future needs. We conducted a paper exercise on 12 products followed by two comprehensive proofs-of-concept (PoCs) with our key application ...