Python Feature Engineering Cookbook

Estimating text complexity by counting sentences

One aspect of a text we can capture in features is its complexity. Usually, longer descriptions that contain multiple sentences spread over several paragraphs tend to provide more information than descriptions with very few sentences. Therefore, capturing the number of sentences may provide some insight into the amount of information provided by the text. This process is called sentence tokenization. Tokenization is the process of splitting a string into a list of pieces or tokens. In the previous Counting characters, words, and vocabulary recipe, we did word tokenization, that is, we divided the string into words. In this recipe, we will divide the string into sentences and then we will count ...