Architecting the Industrial Internet

Data lakes and Hadoop

Hadoop was invented early in this century to enable analysis of data streams common in solving search engine problems. Given that Industrial Internet problems are also solved through analysis of streaming data, Hadoop became an important technology component deployed in many such projects.

Hadoop is supported as an endpoint from IoT and event hubs enabling loading from the speed layer into the batch layer. Events containing data can arrive continuously, and the data is simply appended providing the real-time loading needed for such data volumes.

Hadoop features a utility often used in loading streaming data called Kafka. Some organizations use Kafka to replace traditional message brokers. When Kafka is deployed, producers ...