Webbots, Spiders, and Screen Scrapers, 2nd Edition

Chapter 26. Designing Stealthy Webbots and Spiders

This chapter explores design and implementation considerations that make webbots hard to detect. However, the inclusion of a chapter on stealth shouldn’t imply that there’s a stigma associated with writing webbots; you shouldn’t feel self-conscious about writing webbots, as long as your goals are to create legal and novel solutions to tedious tasks. Most of the reasons for maintaining stealth have more to do with maintaining competitive advantage than covering the tracks of a malicious web agent.

This chapter describes basic stealth principles. The next chapter describes how to achieve anonymity through the use of proxies.

Why Design a Stealthy Webbot?

Webbots that create competitive advantages for ...