Building Google Cloud Platform Solutions

Composite transforms

In order to build higher-level functionality, new Dataflow transforms may be built by extending and combining other transforms, creating composite transforms. Many of the transforms included in the Cloud Dataflow SDK are themselves composite types, such as the MapElements and Count.perKey transforms shown previously. By creating new composite transforms, developers can build reusable pieces of pipeline functionality or even entire libraries to be used across many pipelines.

The examples so far accepted PCollection of strings, extracted words from those strings, mapped words to their first letter, and counted the frequency of each first letter. If this series of transformations occur often, we could encapsulate the entire ...