August 2022
Intermediate to advanced
441 pages
11h 26m
Chinese
Content preview from Python文本分析
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
106
｜
第
4
章
通过本章的学习，你将了解数据准备工作中必要的步骤以及可选的步骤。你将学习
如何使用正则表达式清理数据，以及如何使用
spaCy
提取特征。通过本章案例的学习，
你将掌握在实际项目中快速建立数据处理流水线的技术。
4.2
数据预处理流水线
通常，数据预处理的工作都需要划分成一系列的步骤。我们称这一系列步骤为流水
线，因为你需要将原始数据输入到流水线，然后获得经过转换和预处理后的数据。
在第
1
章中，我们建立了一个简单的数据处理流水线，其中包括分词和删除停用词
的步骤。在本章中，流水线是一系列处理步骤的通称。图
4-1
概述了本章我们将要
构建的预处理流水线。
源文本 识别噪声
剔除噪声
字符规范化
数据遮挡
处理后
的文本
清理后
的文本
分词
词性标注
词形还原
命名实体识别
数据清理
语言处理
图 4-1：包含文本数据预处理常见步骤的流水线
在上述操作中，第一部分主要是数据清理。首先，我们从识别和剔除本文中的噪
声开始，比如
HTML
标签和不可打印字符等。接下来是字符规范化（
character
normalization
），这一步可以将特殊字符（例如重音符号和连字符）转换为标准的
表示形式。最后，我们可以遮挡与分析无关或涉及隐私问题的
URL
或电子邮件地址
之类的标识符。经过这一系列的操作，文本就被清理干净了，然后就可以开始进行
语言处理了。
接下来，分词操作将文档分割成一系列独立的词语（包括单词和标点符号）。词性
标 注（
Part-of-speech tagging
，或 称
POS tagging
）可以确定每个词语的词性，比如名词、 ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.