Flink deployment
Deployment of the Flink pipeline is a simple process. However, it can pose challenges in terms of version mismatch between various libraries. As a rule of thumb, it would be better to reuse the Flink libraries that are available as part of the distribution and then add other libraries which are not available as part of the Flink distribution.
While submitting the JAR as a Flink job, Flink looks for dependencies in its classpath, and if it is not able to find any libraries, it is not able to run the job.
In order to provision all the dependencies in the Flink classpath, it is helpful to copy the shaded jar into ${FLINK_HOME}/lib folder before submitting the JAR as a job for execution. This takes care of all Hadoop and Avro ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access