Data Lake for Enterprises

Flume Channel

A channel is a mechanism used by the Flume agent to transfer data from source to sink. The events are persisted in the channel and until it is delivered/taken away by a sink, they reside in the channel. This persistence in channel allows sink to retry for each event in case there is a failure while persisting data to the real store (HDFS).

Channels can be broadly categorized into two:

1. In-memory: The events are available until the channel component is alive:
   • Queue: In-memory queues in the channel. This has the lowest latency time for processing because the events are persisted in memory.
2. Durable: Even after the component is dead, the event persisted is available, and when the component becomes online, these events will be ...