Distributed Computing with Python

Installing Celery

Celery (http://www.celeryproject.org) is the first third-party library that we encounter in this book, since so far, we have only looked at modules and packages in the Python standard library. Celery is a distributed task queue, meaning that it is a queue-based system like some of the ones that we built in the previous chapters. It is also distributed, which means that worker processes, as well as the queues holding results and work requests, typically run on different machines.

Let's start by installing Celery and its dependencies. We start by setting up a virtual environment on each machine (let's call it book so that we know it is related to the examples in this book), as shown in the following line of code (assuming a Unix ...