Hands-On Industrial Internet of Things

Developing our batch analytics with Airflow

In an I-IoT framework, we need fast data-processing analytics (hot path) and a long-running batch processor to develop our Digital Twin or Deep Learning analytics (cold path). In a previous Kafka Streams as a Rule Engine section, we discovered how to develop a stream-based rule engine in Apache Kafka. Now we want to conclude our experiment to use a batch-processing analytics platform called Apache Airflow. Airflow has good support for the workflow (with particular support for directed acyclic graph (DAG), plugins, and the connector. Airflow can be easily installed on the most important cloud platforms.

In Airflow, a DAG is a collection of all the tasks you want to run, organized in a way that reflects ...