With great power...
As an HTTP client developer, you may have different priorities to the webmasters that run websites. A webmaster will typically provide a site for human users; possibly offering a service designed for generating revenue, and it is most likely that all this will need to be done with the help of very limited resources. They will be interested in analyzing how humans use their site, and may have areas of the site they would prefer that automated clients didn't explore.
HTTP clients that automatically parse and download pages on websites are called various things, such as bots, web crawlers, and spiders. Bots have many legitimate uses. All the search engine providers make extensive use of bots for crawling the web and building their ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access