It is necessary to reduce the size of the feature space of text data by removing unnecessary text that adds noise to the text during analysis. There are a series of steps that are usually performed to preprocess the text data. However, not all steps are required for each task and they are used whenever necessary. For example, if every word in a text data item is already in lower case, there is no need to modify the case of the text to make it uniform.
There are three main elements of a text preprocessing task:
- Tokenization
- Normalization
- Substitution
We will be using the nltk library for demonstrating the different text preprocessing methods. Install the nltk library by issuing the following command in the Command Prompt: ...
