O'Reilly logo

Storm Blueprints: Patterns for Distributed Real-time Computation by Brian O'Neill, P. Taylor Goetz

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Establishing the architecture

We touched on Hadoop in the previous chapter, but we focused mainly on the map/reduce mechanism within Hadoop. In this chapter, we will do the opposite and focus on the Hadoop File System (HDFS) and Yet Another Resource Negotiator (YARN). We will leverage HDFS to stage the data, and leverage YARN to deploy the Storm framework that will host the topology.

The recent componentization within Hadoop allows any distributed system to use it for resource management. In Hadoop 1.0, resource management was embedded into the MapReduce framework as shown in the following diagram:

Establishing the architecture

Hadoop 2.0 separates out resource management into ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required