June 2017
Beginner to intermediate
576 pages
15h 22m
English
In order to normalize the data, we will need the means and standard deviations of each column. If you were doing this purely in base R, there would be many ways to do this using built-in functions and packages. However, using SparkR may limit how we would typically proceed, and force us to be more efficient and process the data in different ways.
Recall that we have already stored this data in a summary object (sumdf_view and so on), so in order to save some processing time, we will simply read the summary statistics back in and avoid having to recalculate the summary statistics.
First, we will read the means row in, followed by the stddev row:
means = sql("select age as age_mean, ...