Data Warehousing in the Age of Big Data

Chapter 3

Big Data Processing Architectures

And pluck till time and times are done The silver apples of the moon, The golden apples of the sun.

—W. B. Yeats

Introduction

Data processing has been a complex subject to deal with since the primitive days of computing. The underlying reason for this stems from the fact that complexity is induced from the instrumentation of data rather than the movement of data. Instrumentation of data requires a complete understanding of the data and the need to maintain consistency of processing (if the data set is broken into multiple pieces), the need to integrate multiple data sets through the processing cycles to maintain the integrity of the data, and the need for complete associated computations within the ...