O'Reilly logo

Clojure High Performance Programming by Shantanu Kumar

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Concurrent pipelines

Imagine a situation where we have to carry out jobs at a certain throughput, such that each job includes the same sequence of a differently sized I/O task (task A), a memory-bound task (task B), and again an I/O task (task C). A naive approach would be to create a thread pool and run each job off it, but soon we realize that this is not optimum because we cannot ascertain the utilization of each I/O resource due to unpredictability of the threads being scheduled by the OS. We also observe that even though several concurrent jobs have similar I/O tasks, we are unable to batch them in our first approach.

As the next iteration, we split each job in to stages (A, B, and C) such that each stage corresponds to one task. Since the ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required