Apache Flume: Distributed Log Collection for Hadoop

Starting up with "Hello World"

No technical book would be complete without a "Hello World" example. Here is the configuration file we'll be using:

agent.sources=s1
agent.channels=c1
agent.sinks=k1
agent.sources.s1.type=netcat
agent.sources.s1.channels=c1
agent.sources.s1.bind=0.0.0.0
agent.sources.s1.port=12345
agent.channels.c1.type=memory
agent.sinks.k1.type=logger
agent.sinks.k1.channel=c1

Here I've defined one agent (called agent) that has a source named s1, a channel named c1, and a sink named k1.

The s1 source's type is netcat, which simply opens a socket listening for events (one line of text per event). It requires two parameters, a bind IP and a port number. In this example we are using 0.0.0.0 for a bind address (the Java convention to ...