Hands-On Big Data Analytics with PySpark

Loading and reloading data about users and followers

To find out which user has which name, we need to load the users.txt file. The users.txt file assigns the VertexId with a username and its own name. We use the following code:

    val users = sc.textFile(getClass.getResource("/pagerank/users.txt").getPath).map { line =>

The following is the users.txt file:

We are splitting on the comma and the first group is our integer, which will be vertex ID, and then fields(1) is the name of vertex, as follows:

      val fields = line.split(",")      (fields(0).toLong, fields(1))    }

Next, we will join the users with ranks. We will join the users using the VertexId ...