Monitoring performance metrics

Now that we have covered a few options for process monitoring, how do you know if your application is actually doing the work you think it is? On many occasions I've seen a stuck syslog-ng process that appeared to be running, but it just wasn't sending any data. I'm not picking on syslog-ng specifically; all software does this when conditions occur that it isn't designed to deal with.

When talking about Flume data flows, you need to monitor the following:

  • Data entering sources is within expected rates
  • Data isn't overflowing your channels
  • Data is exiting sinks at an expected rates

Flume has a pluggable monitoring framework, but as mentioned at the beginning of the chapter, it is still very much a work in progress. That ...

Get Apache Flume: Distributed Log Collection for Hadoop now with the O’Reilly learning platform.

O’Reilly members experience live online training, plus books, videos, and digital content from nearly 200 publishers.