May 2017
Beginner to intermediate
596 pages
15h 2m
English
Content preview from Data Lake for Enterprises
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
Start your free trial
Example 1 - RDBMS to Kafka
As we have created some data with earlier chapters for Sqoop export/import, we will reuse the same data for streaming them as events into Kafka using Flume in this example. Database Source Configuration:
- Copy the PostgresSQL driver jar, downloaded in previous chapters (Chapter 5, Data Acquisition of Batch Data with Apache Sqoop), into ${FLUME_HOME}/lib folder:
cp ${SQOOP_HOME}/lib/postgresql-9.4.1212.jre6.jar$F{LUME_HOME|/lib
- SQL as a source is not a standard source which gets bundled with flume distribution. Hence a third party source needs to be downloaded and installed:
- Download the source from the following location, using this command:
wget https://github.com/keedio/flume-ng-sql-source/archive/1.4.2.tar.gz ...