With great power...
As an HTTP client developer, you may have different priorities to the webmasters that run websites. A webmaster will typically provide a site for human users; possibly offering a service designed for generating revenue, and it is most likely that all this will need to be done with the help of very limited resources. They will be interested in analyzing how humans use their site, and may have areas of the site they would prefer that automated clients didn't explore.
HTTP clients that automatically parse and download pages on websites are called various things, such as bots, web crawlers, and spiders. Bots have many legitimate uses. All the search engine providers make extensive use of bots for crawling the web and building their ...
Get Learning Python Network Programming now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
