Hands-On Software Engineering with Golang

Data-Processing Pipelines

"Inside every well-written large program is a well-written small program."

- Tony Hoare

Pipelines are a fairly standard and used way to segregate the processing of data into multiple stages. In this chapter, we will be exploring the basic principles behind data-processing pipelines and present a blueprint for implementing generic, concurrent-safe, and reusable pipelines using Go primitives, such as channels, contexts, and go-routines.

In this chapter, you will learn about the following:

• Designing a generic processing pipeline from scratch using Go primitives
• Approaches to modeling pipeline payloads in a generic way
• Strategies for dealing with errors that can occur while a pipeline is executing
• Pros and cons of synchronous ...