High Performance Python, 2nd Edition

Chapter 9. The multiprocessing Module

Questions You’ll Be Able to Answer After This Chapter

• What does the multiprocessing module offer?

• What’s the difference between processes and threads?

• How do I choose the right size for a process pool?

• How do I use nonpersistent queues for work processing?

• What are the costs and benefits of interprocess communication?

• How can I process numpy data with many CPUs?

• How would I use Joblib to simplify parallelized and cached scientific work?

• Why do I need locking to avoid data loss?

CPython doesn’t use multiple CPUs by default. This is partly because Python was designed back in a single-core era, and partly because parallelizing can actually be quite difficult to do efficiently. Python gives us the tools to do it but leaves us to make our own choices. It is painful to see your multicore machine using just one CPU on a long-running process, though, so in this chapter we’ll review ways of using all the machine’s cores at once.

Note

We just mentioned CPython—the common implementation that we all use. Nothing in the Python language stops it from using multicore systems. CPython’s implementation cannot efficiently use multiple cores, but future implementations may not be bound by this restriction.

We live in a multicore world—​4 cores are common in laptops, and 32-core desktop configurations are available. If your job can be split to run on multiple CPUs without too much engineering effort, this is a wise direction to consider. ...