August 2022
Intermediate to advanced
441 pages
11h 26m
Chinese
Content preview from Python文本分析
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
105
第 4 章
准备统计和机器学习的
文本数据
从技术的角度来说，任何文本文档都只不过是字符序列。为了针对内容建立模型，
我们需要将文本转换为一系列单词。更通俗地说，这里的“单词”指的是有意义的
字符序列，即词语。但是，仅有词语还不够。想一想单词序列“
New York
”，我们
应该将其视为一个命名实体。正确地识别这些单词序列，看出它们是复合结构，则
需要复杂的语言处理。
通常，数据准备（或称数据预处理）的工作不仅需要将数据转换为适合分析的基础
形式，而且还需要消除干扰噪声。什么是噪声、什么不是噪声，取决于执行何种分析。
在处理文本时，噪声会以各种不同的方式出现。原始的数据可能包含
HTML
标签或
特殊的字符，大多数情况下，我们都应该将其删除。但是出现频率过高的单词往往
不会包含太多含义，也就是我们所说的停用词，它们的存在会在机器学习和数据分
析中引入噪声，因为它们会增加检测模式的难度。
4.1
本章内容概要
在本章中，我们来介绍几个文本预处理流水线的解决方案。这类流水线的输入为原
始文本，它会清理文本，转换并提取文本内容的基本特征。首先，我们介绍使用正
则表达式清理数据和分词，然后再重点介绍如何使用
spaCy
进行语言处理。
spaCy
是一个强大的
NLP
库，拥有现代
API
以及最新的模型。在有些操作中，我们还将使
用
textacy
，这个库提供了一些非常实用的附加功能，特别是在数据预处理方面。此外，
我们还会在必要的时候介绍
NLTK
以及其他库。
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.