Scala and Spark for Big Data Analytics

Getting started with PySpark

Before going deeper, at first, we need to see how to create the Spark session. It can be done as follows:

spark = SparkSession\         .builder\         .appName("PCAExample")\         .getOrCreate()

Now under this code block, you should place your codes, for example:

data = [(Vectors.sparse(5, [(1, 1.0), (3, 7.0)]),),         (Vectors.dense([2.0, 0.0, 3.0, 4.0, 5.0]),),         (Vectors.dense([4.0, 0.0, 0.0, 6.0, 7.0]),)] df = spark.createDataFrame(data, ["features"]) pca = PCA(k=3, inputCol="features", outputCol="pcaFeatures") model = pca.fit(df) result = model.transform(df).select("pcaFeatures") result.show(truncate=False)

The preceding code demonstrates how to compute principal components on a RowMatrix and use them to project the vectors into ...