Practical Real-time Data Processing and Analytics

Flink architecture and execution engine

Flink is a platform for distributed stream and batch data processing. The core of Flink is a streaming dataflow engine. Flink is based on Kappa architecture. Kappa architecture was introduced in 2014 by Jay Kreps in https://www.oreilly.com/ideas/questioning-the-lambda-architecture which addresses the pitfalls of Lambda architecture. So before going into the details of Flink, let's figure out the base of Flink: Kappa architecture. Kappa architecture is designed to handle real-time data processing and continuous data reprocessing using a single data stream. There were two main concerns raised related to Lambda architecture: maintaining two different code bases for real-time analytics and batch analytics ...