Distributed Computing with Python

Celery alternatives – Python-RQ

A lightweight and simpler alternative to Celery is Python-RQ (http://python-rq.org). It is based on Redis alone as a provider of both task queue and result backend. It is intended for those applications where complex task dependencies or task routing is not necessary.

Since Celery and Python-RQ are conceptually very similar, let's jump right in and rewrite one of our earlier examples. Create a new Python script (rq/currency.py) with the following command:

import urllib.request


URL = 'http://finance.yahoo.com/d/quotes.csv?s={}=X&f=p'


def get_rate(pair, url_tmplt=URL):
    # raise Exception('Booo!')

    with urllib.request.urlopen(url_tmplt.format(pair)) as res:
        body = res.read()
    return (pair, float(body.strip()))

This ...