Using Flume

Foreword

The past few years have seen tremendous growth in the development and adoption of Big Data technologies. Hadoop and related platforms are powering the next wave of data analytics over increasingly large amounts of data. The data produced today will be dwarfed by what is expected tomorrow, growing at an ever-increasing rate as the digital revolution engulfs all aspects of our existence. The barrier to entry in this new age of massive data volumes is of course the obvious one: how do you get all this data in your cluster to begin with? Clearly, this data is produced in a wide spectrum of sources spread across the enterprise, and has an interesting mix of interaction, machine, sensor, and social data among others. Any operator who has dealt with similar challenges would no doubt agree that it is nontrivial, if not downright hard, to build a system that can route this data into your clusters in a cost-effective manner.

Apache Flume is exactly built to handle this challenge.

Back in 2011 when Flume went into Incubation at The Apache Software Foundation, it was a project built by Cloudera engineers to address large-scale log data aggregation on Hadoop. Being a popular project from the beginning, it had seen a large number of new requirements ranging from event-ordering to guaranteed-delivery semantics, that came up over its initial releases. Given its popularity and high demand for complex requirements, we decided to refactor the project entirely and make it simpler, more powerful ...