Using Flume

Preface

Today, developers are able to write and deploy applications on a large number of servers in the “cloud” very easily. These applications are producing more data than ever, which when stored and analyzed gives valuable insights that can improve the applications themselves and the businesses that the applications are a part of. The data generated by such applications is often analyzed using systems like Hadoop and HBase.

Analyzing this data is really possible only if you can get the data into these systems from frontend servers. Often, the validity of such analysis becomes less valid as the data becomes older. To get the data into the processing system in near real time, systems like Apache Flume are used. Apache Flume is a system for moving large amounts of data from large numbers of data producers to systems that store, index, or analyze that data. Such systems also decouple the producers from the consumers of the data, making it easy to change either side without the other knowing about it. In addition to decoupling, they also provide failure isolation and an added buffer between the producer and the storage system. The data producers will not know about the storage or indexing system being inaccessible until all of the Flume buffers also fill up—this provides an additional buffer, which might be enough for the storage system to come back online and clear up the backlog of events in the Flume buffers.

In this book, we will discuss in detail why systems like Flume are needed, ...