Clojure Applied

Queues and Workers

Many programs can be seen in whole or in part as task processors, where a task is a unit of work usually mapping to an external request. A web app receives requests to build web pages. A web service receives requests to process API calls. A batch program reads files from disk or a database and processes each one appropriately. All of these common patterns can be modeled as a queue of work farmed out to a pool of workers.

The queue orders and holds the tasks, decoupling the place where work arrives from where it’s processed. The worker pool allows us to create pools of workers with different properties, controlling the amount of concurrency and policies used in managing and monitoring the work. That control allows us to make ...