Data Lake for Enterprises

Data acquisition via Flume into Kafka channel

Now let's use Flume to acquire the address data as well as contacts data from the database and spool file, respectively (same as we did in the previous chapter). In order to achieve this, we will define a single Flume configuration file, ${FLUME_HOME}/conf/customer-address-contact-conf.properties, with dedicated Kafka channels to convert the data of both the sources into events.

The complete Flume configuration is as shown here:

agent.sources = sql-source spool-sourceagent.sources.spool-source.type=spooldiragent.sources.spool-source.spoolDir=<spool-file-data-dir>agent.sources.spool-source.inputCharset=ASCIIagent.sources.sql-source.type=org.keedio.flume.source.SQLSourceagent.sources.sql-source.hibernate.connection.url=jdbc:postgresql://localhost/sourcedb?schema=public ...