Webbots, Spiders, and Screen Scrapers, 2nd Edition

Chapter 22. Scheduling Webbots and Spiders

Up to this point, all of the example webbots have run only when executed directly from a command line or when loaded in a browser. In real-world situations, however, you may want to schedule your webbots and spiders to run automatically and periodically. This chapter describes methods for scheduling webbots to run unattended in a Windows environment. Most readers should have access to the scheduling tool I’ll be using here.

If you are using an operating system other than Windows, don’t despair. Most operating systems support scheduling software of some type. In Unix, Linux, and Mac OS X environments, you can always use the cron command, a text-based scheduling tool. Regardless of the operating system you ...