August 2022
Intermediate to advanced
441 pages
11h 26m
Chinese
Content preview from Python文本分析
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
准备统计和机器学习的文本数据
｜
125
spaCy
使用以下约定：带有下划线的词语属性（如
pos_
）会返回可读
的文本表示形式。不带下划线的
pos
则会返回
spaCy
词性标注的数字
标识符。
注
8
该数字标识符可作为常量导入，例如“
spacy.symbols.
VERB
”。请注意不要混淆二者！
4.6.3
案例：自定义分词
分词是流水线的第一步，而且流水线的一切工作都依赖于正确的词语。
spaCy
的分词器在大多数情况下都可以很好地完成工作，但是它会在
#
、
连字符和下划线处进行分割，有时这会引发问题。因此，可能有必要调整其行为。
我们以下列文字为例进行说明：
text = "@Pete: choose low-carb #food #eat-smart. _url_ ;-) "
doc = nlp(text)
for token in doc:
print(token, end="|")
输出结果：
@Pete|:|choose|low|-|carb|#|food|#|eat|-|smart|.|_|url|_|;-)| | |
spaCy
的分词器完全基于规则。首先，它会按照空格字符分割文本。然后，根据正
则表达式定义的前缀、后缀和中缀拆分规则来进一步拆分其余的词语。同时，还会
使用异常规则来处理语言特有的异常，比如“
can
’
t
”应该拆分成“
ca
”和“
n
’
t
”，
而词元分别为“
can
”和“
not
”。
注
9
如上例所示，
spaCy
的英语分词器包含一个中缀规则：在连字符处进行分割。此外，
它还有一个前缀规则：在“
#
”或“
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.