Data Lake for Enterprises

Flume Setup

Flume configuration will be required for us to stream the contents from DB and the log file into Kafka topics. Here we will follow similar steps to configure both the data sources as discussed in earlier chapters:

1. Make a new Flume configuration in ${FLUME_HOME}/conf, let us call it as customer-data-kafkaChannel-flume-conf.properties. As the name suggests, this would contain all the configurations required to capture customer data.
2. Now let us have a look at the source configuration as shown below:

agent.sources = sql-source spool-sourceagent.sources.spool-source.type=spooldiragent.sources.spool-source.spoolDir=<directory containing the spool file>agent.sources.spool-source.inputCharset=ASCIIagent.sources.sql-source.type=org.keedio.flume.source.SQLSource ...