IPython Interactive Computing and Visualization Cookbook

Distributing Python code across multiple cores with IPython

Despite CPython's GIL, it is possible to execute several tasks in parallel on multi-core computers using multiple processes instead of multiple threads. Python offers a native multiprocessing module. IPython offers an even simpler interface that brings powerful parallel computing features in an interactive environment. We will describe this tool here.

How to do it…

1. First, we launch four IPython engines in separate processes. We have basically two options to do this:
   • Executing ipcluster start -n 4 in a system shell
   • Using the web interface provided in the IPython notebook's main page by clicking on the Clusters tab and launching four engines
2. Then, we create a client that will act as a proxy ...