Learning PySpark

Building the graph

Now that we've imported our data, let's build our graph. To do this, we're going to build the structure for our vertices and edges. At the time of writing, GraphFrames requires a specific naming convention for vertices and edges:

• The column representing the vertices needs to have the name ofid. In our case, the vertices of our flight data are the airports. Therefore, we will need to rename the IATA airport code to id in our airports DataFrame.
• The columns representing the edges need to have a source (src) and destination (dst). For our flight data, the edges are the flights, therefore the src and dst are the origin and destination columns from the departureDelays_geo DataFrame.

To simplify the edges for our graph, we will create ...