Data Lake for Enterprises

Flink deployment

Deployment of the Flink pipeline is a simple process. However, it can pose challenges in terms of version mismatch between various libraries. As a rule of thumb, it would be better to reuse the Flink libraries that are available as part of the distribution and then add other libraries which are not available as part of the Flink distribution.

While submitting the JAR as a Flink job, Flink looks for dependencies in its classpath, and if it is not able to find any libraries, it is not able to run the job.

In order to provision all the dependencies in the Flink classpath, it is helpful to copy the shaded jar into ${FLINK_HOME}/lib folder before submitting the JAR as a job for execution. This takes care of all Hadoop and Avro ...