Data Engineering with Python

Appendix

Building a NiFi cluster

In this book, you have built a Kafka cluster, a ZooKeeper cluster, and a Spark cluster. Instead of increasing the power of a single server, through clustering, you are able to add more machines to increase the processing power of a data pipeline. In this chapter, you will learn how to cluster NiFi so that your data pipelines can run across multiple machines.

In this appendix, we're going to cover the following main topics:

• The basics of NiFi clustering
• Building a NiFi cluster
• Building a distributed data pipeline
• Managing the distributed data pipeline

The basics of NiFi clustering

Clustering in Apache NiFi follows a Zero-Master Clustering architecture. In this type of clustering, there is no pre-defined master. ...