Scala and Spark for Big Data Analytics

Estimator Transformer

An Estimator Transformer transforms the input dataset into the output dataset by first generating a Transformer based on the input dataset. Then the Transformer processes the input data, reading the input column and generating the output column in the output dataset.

Such Transformers are invoked as shown next:

transformer = estimator.fit(inputDF)outputDF = transformer.transform(inputDF)

The examples of Estimator Transformers are as follows:

• IDF
• LDA
• Word2Vec

The diagram of an Estimator Transformer is as follows, where the input column from an input dataset is transformed into an output column generating the output dataset:

In the next few sections, we will look deeper into text analytics using a simple example dataset, ...