Analytics in a Big Data World: The Essential Guide to Data Science and its Applications

CHAPTER 1 Big Data and Analytics

Data are everywhere. IBM projects that every day we generate 2.5 quintillion bytes of data.¹ In relative terms, this means 90 ­percent of the data in the world has been created in the last two years. Gartner projects that by 2015, 85 percent of Fortune 500 organizations will be unable to exploit big data for competitive advantage and about 4.4 million jobs will be created around big data.² Although these estimates should not be interpreted in an absolute sense, they are a strong indication of the ubiquity of big data and the strong need for analytical skills and resources because, as the data piles up, managing and analyzing these data resources in the most optimal way become critical success factors in creating competitive advantage and strategic ­leverage.

Figure 1.1 shows the results of a KDnuggets³ poll conducted during April 2013 about the largest data sets analyzed. The total number of respondents was 322 and the numbers per category are indicated between brackets. The median was estimated to be in the 40 to 50 gigabyte (GB) range, which was about double the median answer for a similar poll run in 2012 (20 to 40 GB). This clearly shows the quick increase in size of data that analysts are working on. A further regional breakdown of the poll showed that U.S. data miners lead other regions in big data, with about 28% of them working with terabyte (TB) size databases.

Figure 1.1 Results from a KDnuggets Poll about Largest Data Sets Analyzed ...