Apache Spark 2.x Cookbook

Dimensionality reduction with singular value decomposition

Often, the original dimensions do not represent data in the best way possible. As we saw in PCA, you can, sometimes, project data to fewer dimensions and still retain most of the useful information.

Sometimes, the best approach is to align dimensions along the features that exhibit the most number of variations. This approach helps eliminate dimensions that are not representative of the data.

Let's look at the following figure again, which shows the best-fitting line on two dimensions:

The projection line shows the best approximation of the original data with one dimension. If we take ...