Kubernetes for Developers

A worker queue example with Python and Celery

Where the CronJob is well positioned to run repeated tasks at a specific schedule, another common need is to process a series of work items more or less constantly. A job is well oriented to running a single task until it is complete, but if the volume of things you need to process is large enough, it may be far more effective to maintain a constant process to work on those items.

A common pattern to accommodate this kind of work uses a message queue, as shown here:

With a message queue, you can have an API frontend that creates the work to be run asynchronously, move that into a queue, and then ...