August 2022
Intermediate to advanced
441 pages
11h 26m
Chinese
Content preview from Python文本分析
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
142
｜
第
5
章
将文档转化成向量的方法有很多种。由于文档向量化是所有机器学习任务的基础，
因此我们需要花费一点功夫来设计和实现自己的向量化器（
vectorizer
）。如果你的
项目需要专用的向量化器，那么可以参考我们的解决方案。
接下来，我们将重点介绍
scikit-learn
实现的两个很流行的模型，即词袋模型及其
TF-IDF
加强版。我们将下载一个大型文档数据集，并使用这些方法将其转化成向量。
可想而知，我们会遇到很多有关数据量与可扩展性的问题。
尽管对于许多复杂的机器学习算法来说，向量化是一项基础技术，但这项技术也可
单独用于计算文档之间的相似性。我们将详细介绍技术本身，并探讨如何对其进行
优化和扩展。有关单词更丰富的表示，请参见第
10
章；有关上下文的讨论，请参见
第
11
章。
通过本章的学习，你将了解如何通过单词或单词组合（称为特征），将文档转换成
数字（向量）。
注
1
我们将尝试多种向量化文档的方法，而你可以根据具体情况决定
使用哪种正确的方法。你将学习为何文档的相似性如此重要，并掌握计算相似性的
标准方法。我们将通过一个示例进行详细说明，展示如何向量化大量的文档，并有
效地计算相似性。
第一小节我们将介绍向量化器的概念，并实际构建一个自己的向量化器。如果你需
要在自己的项目中构建更为复杂的向量化器，则可以参考我们的解决方案。统计单
词的出现次数，并将它们作为向量的方法称为词袋（
bag-of-words
），而且这种方法
已经创建了非常通用的模型。
我们首先介绍一个数据集（内含一百多万条新闻标题），然后介绍一个用例，并在 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.