Java Deep Learning Cookbook

How it works...

In step 1, we used DefaultTokenizerFactory() to create the tokenizer factory to tokenize the words. This is the default tokenizer for Word2Vec and it is based on a string tokenizer, or stream tokenizer. We also used CommonPreprocessor as the token preprocessor. A preprocessor will remove anomalies from the text corpus. The CommonPreprocessor is a token preprocessor implementation that removes punctuation marks and converts the text to lowercase. It uses the toLowerCase(String) method and its behavior depends on the default locale.

Here are the configurations that we made in step 2:

• minWordFrequency(): This is the minimum number of times in which a word must exist in the text corpora. In our example, if a word appears fewer ...