Scala for Machine Learning - Second Edition

Extending Spark

As an open source platform, Apache Spark MLlib can be easily customized, and extended to address specific problems related to requiring the implementation of new statistics, mathematical, or machine learning solutions.

The purpose of this section is to create a Kullback-Leibler divergence as a Spark evaluator. The implementation follows two steps:

• Describe and implement the Kullback-Leibler divergence using Spark 2.0
• Convert the class into a Spark evaluator

Kullback-Leibler divergence

The Kullback-Leibler has been briefly introduced in Chapter 5, Dimension Reduction – Divergences in the context of evaluating the similarity of the frequencies distribution between two datasets. The Kullback-Leibler divergence is a concept borrowed from ...