Java: Data Science Made Easy

Reading the DBLP graph

To start with this project, we first need to read the graph data, and for this, we will use Apache Spark and a few of its libraries. The first library is Spark Data frames, it is similar to R data frames, pandas or joinery, except that they are distributed and based on RDDs. 

Let's read this dataset. The first step is to create a special class Edge for storing the data:

public class Edge implements Serializable {    private final String node1;    private final String node2;    private final int year;    // constructor and setters omitted}

Now, let's read the data:

SparkConf conf = new SparkConf().setAppName("graph").setMaster("local[*]");JavaSparkContext sc = new JavaSparkContext(conf);JavaRDD<String> edgeFile = sc.textFile("/data/dblp/dblp_coauthorship.json.gz"); ...