Distributed Computing with Python

A tour of Celery

What is a distributed task queue and how does Celery implement one? It turns out that distributed task queues are a type of architecture that has been around for quite some time. They are a form of master-worker architecture with a middleware layer that uses a set of queues for work requests (that is, the task queues) and a queue, or a storage area, to hold the results (that is, the result backend).

The master process (also called a client or producer) puts work requests (that is, tasks) into one of the task queues and fetches results from the result backend. Worker processes, on the other hand, subscribe to some or all of the task queues to know what work to perform and put their results (if any) into the result backend.

This is ...