PySpark Cookbook

There's more...

Almost exclusively, every estimator (or, in other words, an ML model) found in the ML module expects to see a single column as an input; the column should contain all the features a data scientist wants such a model to use. The .VectorAssembler(...) method, as the name suggests, collates multiple features into a single column.

Consider the following example:

vectorAssembler = (    feat.VectorAssembler(        inputCols=forest.columns,         outputCol='feat'    ))pca = (    feat.PCA(        k=5        , inputCol=vectorAssembler.getOutputCol()        , outputCol='pca_feat'    ))(    pca    .fit(vectorAssembler.transform(forest))    .transform(vectorAssembler.transform(forest))    .select('feat','pca_feat')    .take(1))

First, we use the .VectorAssembler(...) method to collate all columns ...