O'Reilly logo

Webbots, Spiders, and Screen Scrapers by Michael Schrenk

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Further Exploration

You can expand this basic webbot to do a variety of very useful things. Here is a short list of ideas to get you started on advanced designs.

  • Create a web page with a form that allows people to enter and test the links of any web page.

  • Schedule a link-verification bot to run periodically to ensure that links on web pages remain current. (For information on scheduling webbots, read Chapter 23.)

  • Modify the webbot to send email notifications when it finds dead links. (More information on webbots that send email is available in Chapter 16.)

  • Encase the webbot in a spider to check the links on an entire website.

  • Convert this webbot into a function that is called directly from PHP. (This idea is explored in Chapter 17.)

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required