Deployment with Docker

Container messaging

For the scenario that we just talked about, where back-pressure on individual processing stages causes cascade backflow stoppages, message queues (often alternatively referred to as pub/sub messaging systems) are here to provide us with the exact solution we need. Message queues generally store data as messages in a First-In, First-Out (FIFO) queue structure and work by allowing the sender to add  the desired inputs to a particular stage's queue ("enqueue") and the worker (listener) to trigger on new messages within that queue. When the worker is processing a message, the queue hides it from the rest of the workers and when the worker is complete and successful, the message is removed from the queue permanently. By operating ...