August 2022
Intermediate to advanced
441 pages
11h 26m
Chinese
Content preview from Python文本分析
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
网页抓取与数据提取
｜
103
此外，由于
Scrapy
也使用了
CSS
选择器提取
HTML
内容，因此基本的技术与其他
方法相同。但是，它们的下载方法存在很大差异。采用
Twisted
作为后端产生了一
些开销，而且还建立了一种特殊的编程模型。
你需要仔细考虑一体化的方法是否能够满足你的需求。对于某些网站而言，可能已
有现成的
Scrapy
爬虫，而且还可以重复使用。
3.18
爬虫面临的难题
在抓取内容之前，首先你必须考虑版权和数据保护的问题。
如今使用
React
（
https://reactjs.org
）之类的框架构造的网络应用程序越来越多。这
些应用程序往往只有一页，并通过
API
传输数据。因此，这些网站离开
JavaScript
就无法正常运行了。有的网站会为搜索引擎构建专门的
URL
，这些
URL
也可供爬
虫使用。通常，你可以在
sitemap.xml
中找到相关的信息。你可以试试看关闭浏览器
中的
JavaScript
，然后检查网站是否仍然可以使用。
如果网站需要
JavaScript
，则可以打开浏览器的网页检查器，点击“网络”选项卡，
然后试着点击应用程序的各个地方，就可以捕捉到发送出去的请求。有时，网页会
使用
JSON
传输数据，与
HTML
相比，提取
JSON
通常更容易。然而，你仍然需要
生成各个
JSON
的
URL
，而且有可能还需要其他参数来绕过跨站点请求伪造验证
（
cross-site request forgery
，即
CSRF
）（
https://oreil.ly/_6O_Q
）。
有些请求可能非常复杂，例如
Facebook
、
Instagram ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.