Modern Scala Projects

Step 6 – listing the data preprocessing steps

There is a purpose to preprocessing. In most data analytics tasks, the question that begs to be asked is—is our data necessarily usable? The answer lies in the fact that most real-world datasets need preprocessing, a massaging step meant to give data a new usable form.

With the spam and ham datasets, we identify two important preprocessing steps:

1. Removing punctuation marks:

• Processing punctuation using regular expressions

2. Removing stop words

In the next step, we will write Scala code for two regular expressions, expressions that are fairly simple and only address a small subset of spam. However, it's a start.

In the next step, we will load our datasets into Spark. Naturally, we want a ham ...