August 2022
Intermediate to advanced
441 pages
11h 26m
Chinese
Content preview from Python文本分析
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
准备统计和机器学习的文本数据
｜
139
4.8.2
拼写检查
用户生成的内容存在很多拼写错误，我们可以通过拼写检查器自动更正这些错误。
SymSpell
（
https://oreil.ly/puo2S
）是一款流行的拼写检查器，而且提供了
Python
移
植（
https://oreil.ly/yNs_k
）。然而，根据使用智能手机的经验，我们都知道自动拼写
校正本身可能会引入不合理的词语。因此，你应该仔细检查质量是否真的有所改善。
4.8.3
分词规范化
一般，词语或词语的变体会使用不同的拼写形式，但我们都希望将它们作为同一个
词语进行处理，尤其是统计数量的时候。因此，你应该对这些词语进行规范化，并
将它们映射到一个通用标准。下面是一些例子：
• U.S.A.
或
U.S.
→
USA
• dot-com bubble
→
dotcom bubble
• München
→
Munich
你可以使用
spaCy
的词组匹配器，将这种规范化作为预处理后的步骤集成到流水线
中。如果你不使用
spaCy
，则可以使用简单的
Python
词典将不同的拼写映射成规范
化形式。
4.9
小结和建议
数据项目中经常遇到的一个问题就是：“垃圾进，垃圾出”（
Garbage in, garbage
out
）。对于文本数据尤其如此，因为其内部固有的噪声。因此，数据清理是文本分
析项目中最重要的任务之一。你应该投入足够的精力来确保数据的高质量，并进行
系统检查。在本章中，我们展示了许多解决方案来发现和解决质量的问题。
可靠的分析和健壮的模型的第二个前提条件是规范化。许多文本机器学习算法都基 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.