Data Warehousing in the Age of Artificial Intelligence

Chapter 3. The Data Warehouse Has Changed

The Birth of the Data Warehouse

Decades ago, organizations used transactional databases to run analytics. This resulted in significant stress and hand-wringing by database administrators, who struggled to maintain performance of the application while providing worthwhile insights on the data. New techniques arose, including setting up preaggregated roll-ups or online analytical processing (OLAP) cubes during off-hours in order to accelerate report query performance. The approach was notoriously difficult to maintain and refreshed sporadically, either weekly or monthly, leaving business users in the dark on up-to-date analytics.

New Performance, Limited Flexibility

In the mid-1990s, the introduction of appliance-based data warehouse solutions (Figure 3-1) helped mitigate the performance issues for more up-to-date analytics while offloading the query load on transactional systems. These appliance solutions were optimized transactional databases using column store engines and specialized hardware. Several data warehouse solutions sprang up from Oracle, IBM Netezza, Microsoft, SAP, Teradata, and HP Vertica. However, over time, new challenges arose for appliance-based systems, such as the following:

Usability

Each environment required specialty hardware and software services to set up, configure, and tune.

Cost

Initial investments were high, along with adding capacity for new data sources or general growth. Adding new capacity often resulted ...