O'Reilly logo

Practical Predictive Analytics by Ralph Winters

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Becoming large by starting small

The strategy we will use in this chapter is to first retrieve a small existing publicly available dataset (Pima Indians diabetes). Then we will perform some basic exploratory analysis, compute some key statistical properties, and then use those properties to simulate a much larger dataset that we will use to input into Spark. The key characteristics that we will use to generate this 'big data' will be:

  • The means/standard deviations of the variables: the goal will be to generate means and standard deviations for the large dataset, which are close to the equivalent means and standard deviations of the small dataset.
  • The correlations of the variables: since statistical modeling and analysis is largely based ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required