O'Reilly logo

Apache Spark Graph Processing by Rindra Ramamonjison

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Summary

AggregateMessages provides a functional abstraction for aggregating neighborhood information in Spark graphs. This operator applies a user-defined sendMsg function to each edge in the graph using EdgeContext. Each EdgeContext class accesses the required information about the edge and passes that information to its source node and/or destination node using the sendToSrc and/or sendToDst methods respectively. After all messages have been received by the nodes, the mergeMsg function is used to aggregate those messages at each node.

In the next chapter, we will introduce another operator called Pregel, which will be useful for creating custom iterative graph-processing algorithms.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required