Hands-On Software Engineering with Golang

Using a dead-letter queue

In some scenarios, the user of the pipeline package might be interested in obtaining a list of all the payloads that could not be processed by the pipeline because of the presence of errors.

The following points apply, depending on the application requirements:

• Detailed information about each error and the content of each failed payload can be logged out for further analysis
• Failed payloads can be persisted to an external system (for example, via a messaging queue) so that they can be manually inspected and corrected (when feasible) by human operators
• We could start a new pipeline run to process the payloads that failed during the previous run

The concept of storing failed items for future processing is quite prevalent ...