Apache Flume: Distributed Log Collection for Hadoop

Syslog sources

Syslog has been around for decades and is often used as an operating system level mechanism for capturing and moving logs around systems. In many ways there are overlaps with some of the functionality that Flume provides. There is even a Hadoop module for rsyslog, one of the more modern variants of syslog (http://www.rsyslog.com/doc/rsyslog_conf_modules.html/omhdfs.html). Generally, I don't like solutions that couple technologies that may version independently. If you use this rsyslog/Hadoop integration, you would be required to update the version of Hadoop you compiled into rsyslog at the same time you upgraded your Hadoop cluster to a new major version. This may be logistically difficult if you have a large number of servers ...