Indexing and persisting the log data

Log data needs to be stored for some defined period of time in order to be useful; it also needs to be searchable. In order to achieve this, the recipe integrates with an open source product call Elastic Search, which is a general-use, clustered search engine with a RESTful API (http://www.elasticsearch.org/).

How to do it…

  1. Create a new BaseRichBolt class called IndexerBolt and declare the org.elasticsearch.client.Client client as a private member variable. You must initialize it as follows within the prepare method:
    if((Boolean)stormConf.get(backtype.storm.Config.TOPOLOGY_DEBUG) == true){ node = NodeBuilder.nodeBuilder().local(true).node(); } else { String clusterName = (String) stormConf.get(Conf.ELASTIC_CLUSTER_NAME); ...

Get Storm Real-time Processing Cookbook now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.