Scala and Spark for Big Data Analytics

Reading a dataset in Libsvm format

Let's see how to read data in LIBSVM format using the read API and the load() method by specifying the format of the data (that is, libsvm) as follows:

# Creating DataFrame from libsvm dataset myDF = spark.read.format("libsvm").load("C:/Exp//mnist.bz2")

The preceding MNIST dataset can be downloaded from https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multiclass/mnist.bz2. This will essentially return a DataFrame and the content can be seen by calling the show() method as follows:

myDF.show() 

The output is as follows:

Figure 7: A snap of the handwritten dataset in LIBSVM format

You can also specify ...