Distributed Computing with Python

The tools

In Chapter 3, Parallelism in Python, we looked at a few standard library modules that can be used to introduce (single-node) parallelism in our applications. We experimented with both the threading and multiprocessing modules directly and via the higher-level concurrent.futures module.

We saw how, for non-distributed, parallel applications, Python offers a really robust foundation. The preceding three modules are complete and included in every modern Python distribution. They have no external dependencies, which makes them quite appealing.

We explored a few third-party Python modules for simple distributed computing in Chapter 4, Distributed Applications – with Celery. These included Celery, Python-RQ, and Pyro. We saw how to use them ...