O'Reilly logo

Storm Blueprints: Patterns for Distributed Real-time Computation by Brian O'Neill, P. Taylor Goetz

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Implementing the design

Let's first examine the real-time portion of the system beginning with the spout through to the Druid persistence. The topology is straightforward and mimics topologies we have written in previous chapters.

The following are the critical lines of the topology:

TwitterSpout spout = new TwitterSpout(); Stream inputStream = topology.newStream("nlp", spout); try { inputStream.each(new Fields("tweet"), new TweetSplitterFunction(), new Fields("word")) .each(new Fields("searchphrase", "tweet", "word"), new WordFrequencyFunction(), new Fields("baseline")) .each(new Fields("searchphrase", "tweet", "word", "baseline"), new PersistenceFunction(), new Fields()) .partitionPersist(new DruidStateFactory(), new Fields("searchphrase", "tweet", ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required