Multiple processes

Traditionally, the way Python programmers have worked around the GIL and its effect on CPU-bound threads has been to use multiple processes instead of multiple threads. This approach (multiprocessing) has some disadvantages, which mostly boil down to having to launch multiple instances of the Python interpreter with all the startup time and memory usage penalties that this implies.

At the same time, however, using multiple processes to execute tasks in parallel has some nice properties. Multiple processes have their own memory space and implement a share-nothing architecture, making it easy to reason about data-access patterns. They also allow us to (more) easily transition from a single-machine architecture to a distributed ...

Get Distributed Computing with Python now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.