O'Reilly logo

PySpark Cookbook by Tomasz Drabas, Denny Lee

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Introducing Estimators

The Estimator class, just like the Transformer class, was introduced in Spark 1.3. The Estimators, as the name suggests, estimate the parameters of a model or, in other words, fit the models to data.

In this recipe, we will introduce two models: the linear SVM acting as a classification model, and a linear regression model predicting the forest elevation.

Here is a list of all of the Estimators, or machine learning models, available in the ML module:

  • Classification:
    • LinearSVC is an SVM model for linearly separable problems. The SVM's kernel has the  form (a hyperplane), where  is the coefficients (or a normal vector ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required