Building Google Cloud Platform Solutions

Aggregate transforms

Whereas element-wise transforms operate on the basis of individual units passing through the pipeline, aggregate transforms operate on entire collections for tasks like counting, finding averages, grouping, and flattening. Core aggregate transforms include GroupByKey, CoGroupByKey, Combine, Flatten, and Partition. As with MapElements, the SDK includes several additional transforms that build on these core types, such as Count and Mean. Building on the above example, the frequency of words starting with each letter can be calculated with the high-level aggregate transformation, Count.perKey:

PCollection<KV<Character, Long>> wordsPerCharacter = startingWith .apply(Count.perKey());