Distributed Computing with Python

Multiple processes

Traditionally, the way Python programmers have worked around the GIL and its effect on CPU-bound threads has been to use multiple processes instead of multiple threads. This approach (multiprocessing) has some disadvantages, which mostly boil down to having to launch multiple instances of the Python interpreter with all the startup time and memory usage penalties that this implies.

At the same time, however, using multiple processes to execute tasks in parallel has some nice properties. Multiple processes have their own memory space and implement a share-nothing architecture, making it easy to reason about data-access patterns. They also allow us to (more) easily transition from a single-machine architecture to a distributed ...