Cosine similarity

Word similarity/dissimilarity is measured via cosine similarity, which has a very nice property of being bound between -1 and 1. Perfect similarity between two words will yield a score of 1, no relation will yield 0, and -1 means that they are opposites.

Note that the cosine similarity function for the word2vec algorithm (again, just the CBOW implementation in Spark, for now) is already baked into MLlib, which we will see shortly.

Take a look at the following diagram:

For those who are interested in other measures of similarity, a recent research has been published that makes a strong case for using Earth-Mover's Distance ...

Get Mastering Machine Learning with Spark 2.x now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.