Learn Web Development with Python

Handling failure

All sorts of exceptions can happen while executing a Celery task. In the absence of a well-defined exception handling and retry mechanism, they can go undetected. Often, a job failure is temporary, such as an unresponsive API (which is beyond our control) or running out of memory. In such cases, it is better to wait and retry the task.

In Celery, you can choose to retry automatically or manually. Celery makes it easy to fine-tune its automatic retry mechanism. In the following example, we specify multiple retry parameters:

@shared_task(autoretry_for=(GatewayError,), 
             retry_backoff=60, 
             retry_kwargs={'max_retries': 5}, 
             retry_jitter=True) 
def fetch_feed(feed_id): 
    ... 

The autoretry_for argument lists all the exceptions for which ...