97 Things Every Data Engineer Should Know

Chapter 55. Pipe Dreams

Scott Haines

Senior principal software engineer/software architect

One subtle and novel paradigm in computer systems architecture is the concept of message passing. This simple, albeit useful construct allowed for an order of magnitude of gains in parallel processing by allowing processes (applications, kernel tasks, etc.) within a single computer OS to take part in a conversation. The beauty of this was that processes could now participate in conversations in either synchronous or asynchronous fashion and distribute work among many applications without the necessary overhead of locking and synchronization.

This novel approach to solving parallel processing tasks within a single system was further expanded to distributed systems processing with the advent of the message queue as a service. The basic message queue allowed for one or many channels (or topics) to be created in a distributed first-in, first-out (FIFO) style of queue that could be run as a service on top of a network-addressable location (e.g., ip-address:port). Now many systems across many servers could communicate in a distributed, shared work style, a decomposition of tasks.

It isn’t hard to conceptualize how the pipeline architecture grew out of the concept of distributed systems communicating over network-addressable message queues. It kind of makes sense on a macro level. Essentially, ...