Scala and Spark for Big Data Analytics

Classifying MNIST dataset using random forest

In this section, we will show an example of a classification using the random forest. We will break down the code step-by-step so that you can understand the solution easily.

Step 1. Load and parse the MNIST dataset in LIVSVM format

// Load training data in LIBSVM format. val data = MLUtils.loadLibSVMFile(spark.sparkContext, "data/mnist.bz2")

Step 2. Prepare the training and test sets

Split data into training (75%) and test (25%) and also set the seed for the reproducibility, as follows:

val splits = data.randomSplit(Array(0.75, 0.25), seed = 12345L)val training = splits(0).cache()val test = splits(1)

Step 3. Run the training algorithm to build the model

Train a random forest model with an empty ...