O'Reilly logo

Hands-On Automated Machine Learning by Umit Mert Cakmak, Sibanjan Das

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Text preprocessing

It is necessary to reduce the size of the feature space of text data by removing unnecessary text that adds noise to the text during analysis. There are a series of steps that are usually performed to preprocess the text data. However, not all steps are required for each task and they are used whenever necessary. For example, if every word in a text data item is already in lower case, there is no need to modify the case of the text to make it uniform.

There are three main elements of a text preprocessing task:

  • Tokenization
  • Normalization
  • Substitution

We will be using the nltk library for demonstrating the different text preprocessing methods. Install the nltk library by issuing the following command in the Command Prompt: ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required