O'Reilly logo

Apache Spark 2.x for Java Developers by Sumit Kumar, Sourav Gulati

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Pipelines

The pipeline API is part of the spark.ml package and it consists of a sequence of stages such as data cleaning, Feature extraction, model training, model validation, model testing, and so on. It provides the ability to chain all these different stages in a sequence for developing any machine learning algorithm. First let's get acquainted with common terminologies used in ML workflow using pipelines:

  • Dataframe: Unlike the spark.mllib package, where the datatype was RDD based, in spark.ml the datatype is dataframe. Every pipeline expects input data to be dataframe, which can again be created from various Data sources discussed in Chapter 8, Working with Spark SQL, such as from filesystems, external databases, and other data generation ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required