Google Cloud Platform for Developers

Pipelines

The primary means for interacting with Cloud Dataflow is through pipelines. A pipeline programmatically defines a data processing task from start to finish, taking data from one or more sources, performing a series of transformations on that data, and exporting the results to one or more destinations, or sinks. The Dataflow pipeline model is extremely flexible; pipelines may follow a sequential path from one transformation to the next, or they may be complex with one-to-many, many-to-one, and many-to-many connections between transformations, forming a directed acyclic graph:

Developers define pipelines by writing driver programs ...