Python Natural Language Processing

Indexing

Indexing is quite a useful technique. This is used to convert the categorical data to its numerical format. In an NLP application, you may find that the data attributes are categorical and you want to convert them to a certain numerical value. In such cases, this indexing concept can help you. We can use the SparkML library, which has a variety of APIs to generate indexes. SparkML has an API named StringIndexer that uses the frequency of the categorical data and assigns the index as per the frequency count. So, the most frequent category gets an index value of 0. This can sometimes be a naïve way of generating indexing, but in some analytical applications, you may find this technique useful. You can see the example at this link: ...