Text Processing with Ruby

Concurrency and Buffering

When thinking about pipeline chains, you could be forgiven for thinking that they’re executed in sequence; that is, that the first command generates all of its output, then the second command takes that input and outputs whatever it needs to, and then after that the third process does its bit, and so on.

In reality, though, that’s not the case. All of the programs in the pipeline chain run simultaneously, and data flows between them bit by bit—just like water through a real pipe. While the second process is working with the first chunk of information, the first process is generating another chunk; by the time the first chunk is through to the third or fourth process in the pipeline, the first process may be onto the ...