Daniel Arbuckle's Mastering Python

Calling ProcessPoolExecutor

When we call the ProcessPoolExecutor map or submit methods (which we'll discuss later in this section), we're asking it to call a function with the given parameters. But we want that function call to happen inside a worker process. This has some implications that might not be obvious:

• First of all, it means that the function and its parameters need to be picklable, which is another way of saying that Python needs to know how to turn them into a byte string that it could send to the worker process.

For functions, that basically means any function is OK, unless it was defined within the body of another function.

For the parameters, it means that most objects will work, but generators and a few other kinds of special ...