Big Data Now: 2015 Edition

Chapter 5. The Internet of Things and Real Time

The Internet of Things (IoT) is hot, and as Alistair Croll predicts in "The Internet of Things Has Four Big Data Problems,” it’s here to stay. The abundance of smart devices on the market is generating new questions about how to handle the real-time event data produced downstream. Apache Kafka has emerged as a leader among stream-processing frameworks, with its high throughput, built-in partitioning, replication, and fault tolerance. Numerous other Apache projects, such as Flume and Cassandra, have cropped up and are being used alongside Kafka to effectively collect and store real-time data. Stream processing and data management continues to be an area of intense activity and interest. In this chapter, we recap some of the most exciting advancements in IoT and real time over the past year.

Ben Lorica reviews a few of the more popular components in stream-processing stacks and combinations that are on the rise for collecting, storing, and analyzing event data. John Piekos explores some of the challenges with lambda architecture, and offers an alternative architecture using a fast in-memory scalable relational database that can simplify and extend the capabilities of lambda. Ben Lorica explores how intelligent data platforms are powering smart cities, specifically in Singapore, and highlights the intersection of communities and technologies that power our future cities. Alistair Croll explains why the IoT needs more practical data, ...