Java Data Science Cookbook

Retrieving lemma, part-of-speech, and recognizing named entities from tokens using Stanford CoreNLP

Now that we know how to extract tokens or words from a given text, we will see how we can get different types of information from the tokens such as their lemmas, part-of-speech, and whether the token is a named entity.

The process of lemmatization group inflected forms of a word together so that they can be analyzed as a single text unit. This is similar to the process of stemming with a big difference that stemming does not consider context during its grouping. Therefore, lemmatization is particularly more useful for text data analysis than stemming but requires more computation power.

Part-of-speech tags of the tokens in an article or document ...