Chapter 9: Monitoring Data Pipelines

You now know how to build data pipelines that are production ready—they are idempotent, use atomic transactions, and are version controlled. They are ready to start running in production, but before you can deploy them, you need to be able to monitor them. There will be errors in the code, in the data, in the network, or in other areas outside your control. You will need to know when errors happen, and when things are running as they should.

In this chapter, we're going to cover the following main topics:

  • Monitoring NiFi in the GUI
  • Monitoring NiFi using processors
  • Monitoring NiFi with Python and the REST API

Monitoring NiFi using the GUI

The NiFi GUI provides several ways to monitor your data pipelines. ...

Get Data Engineering with Python now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.