Learning to Love Data Science

Chapter 5. Real-Time Big Data Analytics

Topline Summary

How fast is fast? Are you looking at “real time” from the perspective of a human or a machine? Is there a “time barrier” that puts an upper limit on the efficiency of big data analytics? Those are just a few of the questions keeping data scientists awake at night.

This report, written in late 2012 and early 2013, describes the layers and components of the emerging real-time big data analytics stack, and sketches out the architectures required for applying data science routinely in the real world. 

Oceans of Data, Grains of Time

Imagine that it’s 2007. You’re a top executive at a major search engine company, and Steve Jobs has just unveiled the iPhone. You immediately ask yourself, “Should we shift resources away from some of our current projects so we can create an experience expressly for iPhone users?” Then you begin wondering, “What if it’s all hype? Steve is a great showman…how can we predict if the iPhone is a fad or the next big thing?”

The good news is that you’ve got plenty of data at your disposal. The bad news is that you have no way of querying that data and discovering the answer to a critical question: How many people are accessing my sites from their iPhones?

Back in 2007, you couldn’t even ask the question without upgrading the schema in your data warehouse, an expensive process that might have taken two months. Your only choice was to wait and hope that a competitor didn’t eat your lunch in the meantime. ...