High Performance Python

Chapter 9. The multiprocessing Module

Questions You’ll Be Able to Answer After This Chapter

• What does the multiprocessing module offer?
• What’s the difference between processes and threads?
• How do I choose the right size for a process pool?
• How do I use nonpersistent queues for work processing?
• What are the costs and benefits of interprocess communication?
• How can I process numpy data with many CPUs?
• Why do I need locking to avoid data loss?

CPython doesn’t use multiple CPUs by default. This is partly due to Python’s being designed back in a single-core era, and partly because parallelizing can actually be quite difficult to do efficiently. Python gives us the tools to do it but leaves us to make our own choices. It is painful to see your multicore machine using just one CPU on a long-running process, though, so in this chapter we’ll review ways of using all the machine’s cores at once.

Note

It is important to note that we mentioned CPython above (the common implementation that we all use). There’s nothing in the Python language that stops it from using multicore systems. CPython’s implementation cannot efficiently use multiple cores, but other implementations (e.g., PyPy with the forthcoming software transactional memory) may not be bound by this restriction.

We live in a multicore world—4 cores are common in laptops, 8-core desktop configurations will be popular soon, and 10-, 12-, and 15-core server CPUs are available. If your job can be split to run on multiple CPUs without ...