Mastering Machine Learning with Spark 2.x

Composing all transformations together

In the previous section, we developed individual steps using Spark primitives, that is, UDFs, native Spark algorithms, and H2O algorithms. However, to invoke all these transformation on unseen data requires a lot of manual effort. Hence, Spark introduces the concept of pipelines, mainly motivated by Python scikit pipelines (http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html).

To learn more about the design decision behind Python, we recommend that you read the excellent paper "API design for machine learning software: experiences from the scikit-learn project" by Lars Buitinck et al (https://arxiv.org/abs/1309.0238).

The pipeline is composed of stages that are represented ...