August 2022
Intermediate to advanced
441 pages
11h 26m
Chinese
Content preview from Python文本分析
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
32
｜
第
1
章
很显然，去除语料库中最常见的单词之后，词云可以更好地表达
2015
年的主题，但
是我们仍然可以看到“
today
”和“
challenges
”之类频繁出现却没有太大意义的单词。
我们需要通过某种方法降低这些单词的权重，详情请参见下一节。
1.6.4
案例：
TF-IDF
排名
如图
1-5
所示，可视化最常见的单词通常并不会带给我们太多信息。即
使删除停用词，最常见的单词也常常是领域特定的单词，而且任何数据
子集（切片）中的情况都非常相似。但是，我们想更加突出某个数据切片中出现频
率超出“正常水平”的单词。这里说的切片可以是语料库的任何子集，比如某个发言、
某十年区间的发言或来自某个国家的发言。
我们希望突出显示某一个切片中实际的词频高于总体概率的单词。衡量单词“异常”
因素的算法有很多种。最简单且最佳的方法之一就是使用
TF-IDF
（
term frequency–
inverse document frequency
，即词频逆向文档频率）改善词频。
逆向文档频率
逆向文档频率（inverse document frequency，简称 IDF）是一个加权因子，用
于衡量语料库中词条的“异常”情况。一般，我们用 IDF 来降低普通词条对数
据分析或机器学习的影响。具体来说，首先我们定义词条
t
的文档频率。对于
一个给定的语料库（文档的集合）
C
，
t
的文档频率
df(t)
就是 C 中包含词条
t
的文档数量
d
。用数学公式表达就是：
6 For example, scikit-learn’s TfIdfVectorizer ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.