Java: Data Science Made Easy

Edge features

There are a number of edge features that we can compute: the number of common friends, the total number of distinct friends both people have, and so on.

Let us start with the common friend's feature, which for our problem is the number of common coauthors both authors have. To get them we need to join the edges we selected with all the edges (two times) and then group by the ID and count how many elements are there per group. In SQL, it looks like this:

  SELECT train.id, COUNT(*)    FROM Sample train, Edges e1, Edges e2   WHERE train.node1 = e1.src AND         train.node2 = e2.src AND         e1.dst = e2.dstGROUP BY train.id;

Let's translate this to DataFrame API. First, we use the joins:

Dataset<Row> join = train.join(e1, train.col("node1").equalTo(e1.col("e1_src"))); ...