Chapter 9: Monitoring Data Pipelines
You now know how to build data pipelines that are production ready—they are idempotent, use atomic transactions, and are version controlled. They are ready to start running in production, but before you can deploy them, you need to be able to monitor them. There will be errors in the code, in the data, in the network, or in other areas outside your control. You will need to know when errors happen, and when things are running as they should.
In this chapter, we're going to cover the following main topics:
- Monitoring NiFi in the GUI
- Monitoring NiFi using processors
- Monitoring NiFi with Python and the REST API
Monitoring NiFi using the GUI
The NiFi GUI provides several ways to monitor your data pipelines. ...
Get Data Engineering with Python now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
