Python文本分析

准备统计和机器学习的文本数据

｜

113

显然，上述文本包含许多标签，比如

<lb>

（换行符）和

<tab>

等。我们通过第

1

章

介绍的单词计数，再结合一个简单的正则表达式分词器，检查一下是否还有其他标签：

from blueprints.exploration import count_words

count_words(df, column='text', preprocess=lambda t: re.findall(r'<[\w/]*>', t))

1 The Pandas operations map and apply were explained in “Blueprint: Building a Simple Text Preprocessing

Pipeline” on page 10.

2 Libraries specialized in HTML data cleaning such as Beautiful Soup were introduced in Chapter 3.

characters. However, it doesn’t need to be perfect; it just needs to be good enough to

indicate potential quality issues.

For the Reddit data, we can get the most “impure” records with the following two

statemen ...