Internet of Things and Data Analytics Handbook

22HADOOP TECHNOLOGY

SCOTT SHAW

Hortonworks, Inc., Santa Clara, CA, USA

22.1 INTRODUCTION

We are in the midst of a transformation in how we collect, process, analyze, and make use of data. Data is quickly becoming the new standard defining corporate success and competitive differentiation. Those companies who fail to make use of all their data risk falling behind to their competitors who gain from their data key metrics on customers and transactions. Furthermore, the new business models include data aggregators whose business exists because they have been able to collect data from a potentially unlimited number of sources and analyze the full spectrum of the data set to derive never‐before‐seen insights.

Of course, collecting, storing, and processing data comes at a steep cost as well as deep complexity. Since the advent of what we consider the modern computer, the means of storing data relationally or in data warehouses (essentially relational databases logically restructured to provide faster query access for unique analytic workloads) was the primary means to serve data to business analysts. What it took to begin shifting us away from the traditional approach was really big problems involving really big data of which only Internet could cause. Companies such as Google and Yahoo faced a change or die event, forcing them to completely rethink how they store and process data. Hadoop is the outcome of this event and, though it was built for a specific use case, it has emerged ...