May 2017
Beginner to intermediate
596 pages
15h 2m
English
Content preview from Data Lake for Enterprises
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
Start your free trial
Apache Flink
Apache Flume is used mainly for data acquisition capability. We will be using Flume to transfer data from source systems sending stream data to the messaging layer (for further processing) and all the way into HDFS.
For transferring data all the way to HDFS, Apache Flume is best fit for stream data. However for getting stream data and then processing is one of the main use case for Apache Flink and it does have additional features suited for this.
This doesn't mean that Apache Flink can be used for transferring data to HDFS, it does have the mechanism but there willn't be so many built-in capabilities. It does have many features as against Flume but they are more on the stream processing aspects. Flink does have a rolling file ...