Apache Spark 2.x Cookbook

How to do it...

1. Load pres.csv to HDFS:

        $ hdfs dfs -put pres.csv 

2. Start the Spark shell:

        $ spark-shell

3. Import the statistics and related classes:

        scala> import org.apache.spark.mllib.linalg.Vectorsscala> import org.apache.spark.mllib.linalg.distributed.RowMatrix

4. Load pres.csv as an RDD:

        scala> val data = sc.textFile("pres.csv")

5. Transform data into an RDD of dense vectors:

        scala> val parsedData = data.map( line =>         Vectors.dense(line.split(',').map(_.toDouble)))

6. Create RowMatrix from parsedData:

        scala> val mat = new RowMatrix(parsedData)

7. Compute svd:

        scala> val svd = mat.computeSVD(2,true)

8. Calculate the U factor (eigenvector):

        scala> val U = svd.U

9. Calculate the matrix of singular values (eigenvalues):

        scala> ...