Hands-On Big Data Analytics with PySpark

Computing Pearson and Spearman correlations

To understand this, let's assume that we are taking the first three numeric variables from our dataset. For this, we want to access the csv variable that we defined previously, where we simply split raw_data using a comma (,). We will consider only the first three columns that are numeric. We will not take anything that contains words; we're only interested in features that are purely based on numbers. In our case, in kddcup.data, the first feature is indexed at 0; feature 5 and feature 6 are indexed at 4 and 5, respectively which are the numeric variables that we have. We use a lambda function to take all three of these into a list and put it into the metrics variable:

metrics = csv.map(lambda ...