Skip to Main Content
Spidering Hacks
book

Spidering Hacks

by Morbus Iff, Tara Calishain
October 2003
Beginner to intermediate content levelBeginner to intermediate
428 pages
11h 9m
English
O'Reilly Media, Inc.
Content preview from Spidering Hacks

Chapter 1. Walking Softly

Hack 1. Hacks #1-7

With over three billion pages on the Web, serious surfers eventually find themselves asking two questions: where’s the good stuff and what can I do with it? Everyone has their own idea of what the “good stuff” is, and most people come up with some creative idea of what to do once they find it. In some corners of the Web, repurposing data in interesting ways is encouraged: it inspires those “Eureka!” moments when unusual information combinations bubble forth unimpeded.

From the Web’s standpoint, the utility of universally accessible data has only recently been broached. Once Google opened their search listings via an API (see Google Hacks), Amazon.com quickly followed (see Amazon Hacks), and both have benefited by the creative utilities that have resulted. In this short and sweet chapter, we’ll introduce you to the fine art of scraping and spidering: what they are and aren’t, what’s most likely allowed and what might create risk, finding alternative avenues to your desired data, and how to reassure—and, indeed, educate—webmasters who spot your automation and wonder what you’re up to.

Hack #1. A Crash Course in Spidering and Scraping

A few of the whys and wherefores of spidering and scraping.

There is a wide and ever-increasing variety of computer programs gathering and sifting information, aggregating resources, and comparing data. Humans are just one part of a much larger and automated equation. But despite the variety of programs out there, ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

QuickBooks 2005: The Missing Manual

QuickBooks 2005: The Missing Manual

Bonnie Biafore

Publisher Resources

ISBN: 0596005776Supplemental Content