O'Reilly logo

Practical Predictive Analytics by Ralph Winters

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Normalizing the data

We now have all the needed statistics to normalize the data. Recall that the formula for normalizing a variable x is as follows:

In order to implement this, we will wrap the needed computations into a function and invoke it for both the training and test datasets:

  • Use the SparkR selectExpr expression to calculate the normalized version of each variable using the formula above.
  • Also, create a new variable with old appended to the name, which preserves the original value of the variable. After testing, you should remove these extra variables to save space, but it is good to retain them while debugging:
 normalize_it <- ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required