Natural Language Processing with Spark NLP

Chapter 17. Supporting Multiple Languages

When building an NLP system, the first thing you should answer is what language or languages will you support. This can affect everything from data storage, to modeling, to the user interface. In this chapter, we will talk about what you want to consider if you are productionizing a multilingual NLP system.

At the end of the chapter, we will have a checklist of questions to ask yourself about your project.

Language Typology

When supporting multiple languages, one way you can manage complexity is by identifying commonalities between your expected languages. For example, if you are dealing with only Western European languages, you know that you need to consider only the Latin alphabet and its extensions. Also, you know that all the languages are fusional languages, so stemming or lemmatizing will work. They also have similar grammatical gender systems: masculine, feminine, and maybe an inanimate neuter.

Let’s look at a hypothetical scenario.

Scenario: Academic Paper Classification

In this scenario, your inputs will be text documents, PDF documents, or scans of text documents. The output is expected to be JSON documents with text, title, and tags. The languages you will be accepting as input are English, French, German, and Russian. You have labeled data, but it is from only the last five years of articles. This is when the publisher started requiring that articles be tagged during submission. The initial classifications can be at the ...