Data Lake for Enterprises

History of Flume

Apache Flume was developed by Cloudera for handling and moving large amount data produced into Hadoop. Without minimum or no delay (NRT: Near Real Time or Real time) the company wanted the data produced to be moved to Hadoop system, for various analysis to be carried. That was how this beautiful came into existence.

As detailed in previous section, it was initially conceived and developed to take care of a particular use case of collecting and aggregating log data from various source (web servers) into Hadoop for performing various analytics useful for proactive maintenance. Later on it was redesigned and refactored to include different sources and destinations and also design was taken into account for pluggability and extensibility ...