Scala and Spark for Big Data Analytics

Spark ML

Spark ML adds a new set of machine learning APIs to let users quickly assemble and configure practical machine learning pipelines on top of datasets. Spark ML aims to offer a uniform set of high-level APIs built on top of DataFrames rather than RDDs that help users create and tune practical machine learning pipelines. Spark ML API standardizes machine learning algorithms to make the learning tasks easier to combine multiple algorithms into a single pipeline or data workflow for data scientists. The Spark ML uses the concepts of DataFrame and Datasets, which are much newer concepts introduced (as experimental) in Spark 1.6 and then used in Spark 2.0+.

In Scala and Java, DataFrame and Dataset have been unified, that is, DataFrame is ...