Managing and Mitigating Bots

Chapter 5. Data Harvesting

This category captures a range of traffic types that will access the publicly available information contained within your website and capture that data for use elsewhere. Typically this will involve accessing many pages and extracting relevant data using text pattern matching.

Good or Bad?

Data harvesting covers the full range of motives, from good to bad.

Some data harvesting bots, such as search engine spiders, are clearly usually regarded as good, with many businesses depending on these as the source of their traffic. Likewise, affiliates would typically be driving traffic to your site.

Conversely, there are data harvesting bots that are clearly bad, such as those engaged in content theft or price scraping.

There are also data harvesting bots that exist in the gray area between good and bad—for example, price and content aggregators that may be legitimately driving traffic to your site but in a way that may not be in the interests of your business.

Let’s now look at some specific examples of data harvesting bots.

Search Engine Spiders

The most common form of data harvesting and the one without which the internet as we know it today wouldn’t function is the search engine spider. The most common of which is, of course, GoogleBot, but there are many others from a range of global as well as regional or specialist search engines. These bots will usually enter your site via the homepage or via a deep link from another site and then follow all active ...