O'Reilly logo

Machine Learning with Spark - Second Edition by Nick Pentreath, Manpreet Singh Ghotra, Rajdeep Dua

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Singular values

Singular values lets us understand the trade-off between space and time for fidelity of the reduction.

As with evaluating values of k for clustering, in the case of SVD (and PCA), it is often useful to plot the singular values for a larger range of k, and see where the point on the graph is where the amount of additional variance accounted for by each additional singular value starts to flatten out considerably.

We will do this by first computing the top 300 singular values, as shown next:

val svd300 = matrix.computeSVD(300, computeU = false) val sMatrix = new DenseMatrix(1, 300, svd300.s.toArray) println(sMatrix) csvwrite(new File("/home/ubuntu/work/ml-resources/  spark-ml/Chapter_09/data/s.csv"), sMatrix)

We will write ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required