Blueprints for Text Analytics Using Python

Chapter 4. Preparing Textual Data for Statistics and Machine Learning

Technically, any text document is just a sequence of characters. To build models on the content, we need to transform a text into a sequence of words or, more generally, meaningful sequences of characters called tokens. But that alone is not sufficient. Think of the word sequence New York, which should be treated as a single named-entity. Correctly identifying such word sequences as compound structures requires sophisticated linguistic processing.

Data preparation or data preprocessing in general involves not only the transformation of data into a form that can serve as the basis for analysis but also the removal of disturbing noise. What’s noise and what isn’t always depends on the analysis you are going to perform. When working with text, noise comes in different flavors. The raw data may include HTML tags or special characters that should be removed in most cases. But frequent words carrying little meaning, the so-called stop words, introduce noise into machine learning and data analysis because they make it harder to detect patterns.

What You’ll Learn and What We’ll Build

In this chapter, we will develop blueprints for a text preprocessing pipeline. The pipeline will take the raw text as input, clean it, transform it, and extract the basic features of textual content. We start with regular expressions for data cleaning and tokenization and then focus on linguistic processing with spaCy. spaCy is a powerful ...