Cosine similarity

Word similarity/dissimilarity is measured via cosine similarity, which has a very nice property of being bound between -1 and 1. Perfect similarity between two words will yield a score of 1, no relation will yield 0, and -1 means that they are opposites.

Note that the cosine similarity function for the word2vec algorithm (again, just the CBOW implementation in Spark, for now) is already baked into MLlib, which we will see shortly.

Take a look at the following diagram:

For those who are interested in other measures of similarity, a recent research has been published that makes a strong case for using Earth-Mover's Distance ...

Get Mastering Machine Learning with Spark 2.x now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.