September 2013
Intermediate to advanced
264 pages
5h
Chinese
Content preview from Bad Data 技術手冊
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
一般工作流程範例
|
75
robots.txt
幾乎所有的網站都有一個
robots.txt
檔案，它會告訴網頁爬蟲（也就是你）哪些目錄可以
被抓取、它多常可以被拜訪以及其他的事項。列在
robots.txt
中的規則並沒有任何法律責
任，但從 90 年代中期大家漸漸開始會遵守。你也應該要確認網站的服務條款（terms of
service），這就會有法律責任。
這是一個在 App Annie 上的範例
robots.txt
檔案，它是一個聚集並且回報在 iOS App
Store 中對 App 評論的網站。
User-agent: *
Crawl-delay: 5
Sitemap: http://www.appannie.com/media/sitemaps/sitemapindex.xml
Disallow: /top/*date=
Disallow: /matrix/*date=
Disallow: /search/*
Disallow: /search_ac/
Disallow: */ranking_table/
Disallow: /top-table/*
Disallow: /matrix-table/*
Disallow: */ranking/history/
Disallow: */ranking/history/chart_data/
讓我們一部份一部份的來看吧。
User-agent: *
以下的規則套用在所有人之上。有時網站會允許或是阻擋特定的網站爬蟲，如
Google 的索引器。
Crawl-delay: ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.