Skip to Main Content
Hands-On Natural Language Processing with PyTorch 1.x
book

Hands-On Natural Language Processing with PyTorch 1.x

by Thomas Dop
July 2020
Beginner to intermediate content levelBeginner to intermediate
276 pages
6h 5m
English
Packt Publishing
Content preview from Hands-On Natural Language Processing with PyTorch 1.x

Chapter 4: Text Preprocessing, Stemming, and Lemmatization

Textual data can be gathered from a number of different sources and takes many different forms. Text can be tidy and readable or raw and messy and can also come in many different styles and formats. Being able to preprocess this data so that it can be converted into a standard format before it reaches our NLP models is what we'll be looking at in this chapter.

Stemming and lemmatization, similar to tokenization, are other forms of NLP preprocessing. However, unlike tokenization, which reduces a document into individual words, stemming and lemmatization are attempts to reduce these words further to their lexical roots. For example, almost any verb in English has many different variations, ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Hands-On Natural Language Processing with Python

Hands-On Natural Language Processing with Python

Rajesh Arumugam, Rajalingappaa Shanmugamani

Publisher Resources

ISBN: 9781789802740Supplemental Content