Natural Language Processing with Spark NLP

Chapter 9. Information Extraction

When working with text, we generally want to extract some meaning. Many tasks involve extracting events or entities from the text. We may also want to find all the places in a document that reference a particular person. We may want to find what happened in a particular place. These tasks are called information extraction.

Information extraction tasks are focused on entities (nouns and noun phrases) and events (verb phrases including their subjects and objects). This is different than part-of-speech tasks in which, instead of needing to tag everything, we need to identify only the “important” pieces of the text. However, the techniques used to extract this information are generally the same as those used in sequence modeling.

By far, the most common type of information extraction is named-entity recognition. This is the task of finding references to specific entities.

Named-Entity Recognition

Named-entity recognition (NER) is the task of finding specific things (nouns) in text. Often, the desired named entities are proper nouns, but there are other things we may wish to extract as well. Let’s look at some of the common types of nouns that are extracted.

To fully understand the linguistics behind named-entity recognition, we need to define some terms:

Referring expression or R-expression

A word or phrase that refers to an actual or conceptual thing. This is broken into different types based on how specific the R-expression is and how it is specified. ...