Learning Python Networking - Second Edition

Celery architecture

Celery is an efficient and scalable way to execute tasks asynchronously and acts as a distributor of high-level tasks where tasks are queued and run concurrently using various paradigms such as multiprocessing or gevent.

An efficient and scalable way to perform tasks asynchronously is to use a queue library as Celery. With this library, you define workers that are processes for executing the heavy tasks. An interesting aspect of this solution is that there can be many workers (even in different servers) executing the tasks.

The architecture of the solution is as follows:

• Consumer: This is the application that users use. If it is a web application, it can be a Django or Flask application.
• Producer: This is the worker who ...