O'Reilly logo

Apache Flume: Distributed Log Collection for Hadoop by Steve Hoffman

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 3. Channels

In Flume, a channel is the construct used between sources and sinks. It provides a holding area for your in-flight events after they are read from sources until they can be written to sinks in your data processing pipelines.

The two types we'll cover here are a memory-backed/non-durable channel and a local filesystem backed/durable channel. The durable file channel flushes all changes to disk before acknowledging receipt of the event to the sender. This is considerably slower than using the non-durable memory channel, but provides recoverability in the event of system or Flume agent restarts. Conversely, the memory channel is much faster, but failure results in data loss and has much lower storage capacity when compared with ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required