Yahoo! Hacks

Spider the Yahoo! Catalog

Writing a spider to spider an existing spider’s site may seem convoluted, but it can prove useful when you’re looking for location-based services. This hack walks through creating a framework for full-site spidering, including additional filters to lessen your load.

In this hack, you’ll learn how to write a spider that crawls the Yahoo! group of portals. The choice of Yahoo! was obvious; because it is one of the largest Internet portals in existence, it can serve as an ideal example of how one goes about writing a portal spider.

But before we get to the gory details of code, let’s define what exactly a portal spider is. While many may argue with such a classification, I maintain that a portal spider is a script that automatically downloads all documents from a preselected range of URLs found on the portal’s site or a group of sites, as is the case with Yahoo!. A portal spider’s main job is to walk from one document to another, extract URLs from downloaded HTML, process said URLs, and go to another document, repeating the cycle until it runs out of URLs to visit. Once you create code that describes such basic behavior, you can add additional functionality, turning your general portal spider into a specialized one.

Although writing a script that walks from one Yahoo! page to another sounds simple, it isn’t, because there is no general pattern followed by all Yahoo! sites or sections within those sites. Furthermore, Yahoo! is not a single site with a nice link ...