Modern Python Standard Library Cookbook

Sharing data between processes

When working with threads or coroutines, data is shared across them by virtue of the fact that they share the same memory space. So, you can access any object from any thread, as long as attention is paid to avoiding race conditions and providing proper locking.

With processes, instead, things get far more complicated and no data is shared across them. So when using ProcessPool or ProcessPoolExecutor, we need to find a way to pass data across the processes and make them able to share a common state.

The Python standard library provides many tools to create a communication channel between processes: multiprocessing.Queues, multiprocessing.Pipe, multiprocessing.Value, and multiprocessing.Array can be used to create ...