There is no better way to gain a deeper understanding of the BSP model principles than to build, from scratch, our very own scalable Pregel-like graph processing system in Go.
Here are a few of the design requirements for the system we will be building:
- Graphs will be represented as a collection of vertices and directed edges. Each vertex will be assigned a unique ID. In addition, both vertices and edges can optionally store a user-defined value.
- At every super-step, the system executes a user-defined compute function for every vertex in the graph.
- Compute functions are allowed to inspect and modify the internal state of the vertex they are invoked on. They can also iterate the list of outgoing edges ...