O'Reilly logo

Mastering matplotlib by Duncan M. McGreggor

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

The custom ZeroMQ cluster

In this section, we will create a task pipeline, which is a messaging pattern that we mentioned briefly when discussing the Disco project's answer to MapReduce. Task pipelines can be viewed as a generalization of MapReduce in that they support data flows through a sequence of steps, where each step provides the results from its processing to the next step in the flow. We will accomplish this by using ZeroMQ to create a messaging topology that is suitable for the execution of embarrassingly parallel code in a number of worker processes.

Note

The descriptive term embarrassingly parallel was adopted in online parallelization discussions after its use in an article named Matrix Computation on Distributed Memory Multiprocessors ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required