Airflow
As an effective ML practitioner, you will need to programmatically handle workflows such as the previous one and be able to automate them, as well. Airflow provides you with a platform to efficiently do this. This link—https://airflow.apache.org—is an excerpt taken from Airflow's official website. Airflow is a platform used to programmatically author, schedule, and monitor workflows.
The main advantage of this is that tasks represented on Directed Acyclic Graphs (DAGs) can easily be distributed across available resources (often known as workers). It also makes it easier to visualize your entire workflow and this turns out to be very helpful, especially when a workflow is very complicated. If you need a refresher on DAGs, the article ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access