August 2022
Intermediate to advanced
441 pages
11h 26m
Chinese
Content preview from Python文本分析
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
准备统计和机器学习的文本数据
｜
127
修改分词的过程中需要多加小心，因为它们的效果可能会很微妙，针对
某一组情况修改分词的时候可能会破坏其他情况。例如，经过修改之后，
“
Chicago-based
”等词语不会再被拆分。另外，连字符和破折号有好
几个
Unicode
版本，如果不进行规范化，就可能会引发问题。
4.6.4
案例：处理停用词
spaCy
在完成分词后，会立即使用特定于语言的停用词列表，设定每个
词语的
is_stop
属性。这样可以方便过滤停用词（以及类似的标点符号）：
text = "Dear Ryan, we need to sit down and talk. Regards, Pete"
doc = nlp(text)
non_stop = [t for t in doc if not t.is_stop and not t.is_punct]
print(non_stop)
输出结果：
[Dear, Ryan, need, sit, talk, Regards, Pete]
你只需导入
spacy.lang.en.STOP_WORDS
，就可以获得包含
300
多个条目的英语停用
词列表。在创建
nlp
对象时，该列表就会加载，并存储到
nlp.Defaults.stop_words
下。我们可以修改
spaCy
的默认行为，具体方法是设置
spaCy
词汇表中各个单词的
is_stop
属性：
注
11
nlp = spacy.load('en_core_web_sm')
nlp.vocab['down'].is_stop ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.