Expert Python Programming - Third Edition

Using process pools

Using multiple processes instead of threads adds some substantial overhead. Mostly, it increases the memory footprint because each process has its own and independent memory context. This means allowing unbound numbers of child processes is even more of a problematic issue than it is in multithreaded applications.

The best pattern to control resource usage in applications that rely on multiprocessing is to build a process pool in a similar way to what we described for threads in the Using thread pool section.

And the best thing about the multiprocessing module is that it provides a ready to use Pool class that handles all the complexity of managing multiple process workers for you. This pool implementation greatly reduces ...