Data Lake for Enterprises

Step 2 - Consuming Streams via Flink

Now that we have all 100K records queued into Kafka, the next step is to consume these messages using Flink and start establishing an execution pipeline within Flink.

Flink comes with a lot of inbuilt connectors, and one of the source connectors is the Kafka connector. In order to include the Kafka connector, the following dependency is required to be added into the project’s pom.xml file (refer to flink-example1/pom.xml):

<dependency>   <groupId>org.apache.flink</groupId>   <artifactId>flink-connector-kafka-0.10_2.10</artifactId   <version>1.2.0</version></dependency><dependency>   <groupId>org.apache.flink</groupId>   <artifactId>flink-streaming-java_2.11</artifactId   <version>1.2.0</version></dependency>

Code ...