In general, there is no reason to have more processes than there are processors on the computer. There are a few reasons for this:
- Only cpu_count() processes can run simultaneously
- Each process consumes resources with a full copy of the Python interpreter
- Communication between processes is expensive
- Creating processes takes a non-zero amount of time
Given these constraints, it makes sense to create at most cpu_count() processes when the program starts and then have them execute tasks as needed. This has much less overhead than starting a new process for each task.
It is not difficult to implement a basic series of communicating processes that does this, but it can be tricky to debug, test, and get right. Of course, ...
