August 2022
Intermediate to advanced
441 pages
11h 26m
Chinese
Content preview from Python文本分析
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
120
｜
第
4
章
4.5.2 NLTK
分词
下面，我们来看一看
NLTK
的分词器，因为
NLTK
是常用的分词方式。我们可以通
过快捷方式
word_tokenize
调用标准的
NLTK
分词器。应用到我们的示例文本后，
可以得出以下结果：
import nltk
tokens = nltk.tokenize.word_tokenize(text)
print(*tokens, sep='|')
输出结果：
2019-08-10|23:32|:|@|pete/|@|louis|-|I|do|n't|have|a|well-designed
solution|for|today|'s|problem|.|The|code|of|module|AC68|should|be|-1|.
Have|to|think|a|bit|...|#|goodnight|;|-|)||
该函数内部结合使用了
TreebankWordTokenizer
和
PunktSentenceTokenizer
。它
可以处理标准文本，但在处理主题标签或文本表情符号方面存在缺陷。
NLTK
还
提供了
RegexpTokenizer
，该函数基本上就是
re.findall()
的包装，还附带一
些便捷的功能。除此之外，
NLTK
中还有其他基于正则表达式的分词器，例如
TweetTokenizer
以及多语言的
ToktokTokenizer
，详情请参见
GitHub
中本章的
notebook
（
https://oreil.ly/zLTEl
）。
4.5.3
分词建议
如果你希望在特定领域的词语模式上实现准确度非常高的分词，则可能需要使用自 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.