O'Reilly logo

Practical Predictive Analytics by Ralph Winters

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Inputting the previously computed means and standard deviations

In order to normalize the data, we will need the means and standard deviations of each column. If you were doing this purely in base R, there would be many ways to do this using built-in functions and packages. However, using SparkR may limit how we would typically proceed, and force us to be more efficient and process the data in different ways.

Recall that we have already stored this data in a summary object (sumdf_view and so on), so in order to save some processing time, we will simply read the summary statistics back in and avoid having to recalculate the summary statistics.

First, we will read the means row in, followed by the stddev row:

 means = sql("select age as age_mean, ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required