Building Data Streaming Applications with Apache Kafka

Level of parallelism

Any stream processing engine of your choice has ways to tune stream processing parallelism. You should always give a thought to the level of parallelism required for your application. A key point here is that you should utilize your existing cluster to its maximum potential to achieve low latency and high throughput. The default parameters may not be appropriate as per your current cluster capacity. Hence, while designing your cluster, you should always come up with the desired level of parallelism to achieve your latency and throughput SLAs. Moreover, most of the engines are limited by their automatic ability to determine the optimal number of parallelism.

Let’s take Spark's processing engine as an example and see how ...