Streaming Analytics with IBM Streams: Analyze More, Act Faster, and Get Continuous Insights

1Big Data: At Rest and in Motion

Real-time streaming analytics has become an important subject over the last few years. Despite its importance, however, the topic is still shrouded in some misconceptions, partly due to the confusion between data-at-rest processing and data-in-motion processing. In addition, data in motion does not live in a vacuum. It is part of an overall solution for the big data challenge. Before we can dive into streaming analytics, we need to put it in context with a general discussion on big data.

This chapter takes a look at the origins of big data and talks about the different technologies used to solve the big data challenge. Since a big data solution must consider existing business solutions, we also look at the role of relational databases in this environment. We continue by looking at what is needed in a big data architecture followed by the need for data-in-motion processing.

Where Does Big Data Come From?

One could say that big data has been around forever. In information management, there were always challenges due to the amount of data people wanted to analyze and what the technology could support at the time. The amount of data has always been increasing with hardware capacity. Computer memory went from kilobytes to megabytes and now terabytes. The same happened with disk technology. CPU power doubled roughly every 18 months and memory and disk capacity doubled every two years. So, what changed?

You could arguably trace the origin of big data to ...