Distributed Computing with Python

Multiprocess queues

When using multiple processes, the issue that comes up is how to exchange data between the workers. The multiprocessing module offers a mechanism to do that in the form of queues and pipes. Hence, we are going to look at multiprocess queues.

The multiprocessing.Queue class is modeled after the queue.Queue class with the additional twist that items stored in the multiprocessing queue need to be pickable. To illustrate how to use these queues, create a new Python script (queues.py) with the following code:

import multiprocessing as mp


def fib(n):
    if n <= 2:
        return 1
    elif n == 0:
        return 0
    elif n < 0:
        raise Exception('fib(n) is undefined for n < 0')
    return fib(n - 1) + fib(n - 2)


def worker(inq, outq):
    while True:
 data ...