Fast Data: Smart and at Scale

Chapter 4. Recipe: Design Data Pipelines

Idea in Brief

Processing big data effectively often requires multiple database engines, each specialized to a purpose. Databases that are very good at event-oriented real-time processing are likely not good at batch analytics against large volumes. Some systems are good for high-velocity problems. Others are good for large-volume problems. However, in most cases, these systems need to interoperate to support meaningful applications.

Minimally, data arriving at the high-velocity, ingest-oriented systems needs to be processed and captured into the volume-oriented systems. In more advanced cases, reports, analytics, and predictive models generated from the volume-oriented systems need to be communicated to the velocity-oriented system to support real-time applications. Real-time analytics from the velocity side need to be integrated into operational dashboards or downstream applications that process real-time alerts, alarms, insights, and trends.

In practice, this means that many big data applications sit on top of a platform of tools. Usually the components of the platform include at least a large shared storage pool (like HDFS), a high-performance BI analytics query tool (like a columnar SQL system), a batch processing system (MapReduce or perhaps Spark), and a streaming system. Data and processing outputs move between all of these systems. Designing that dataflow—designing a processing pipeline—that coordinates these different platform ...