Managing and Mitigating Bots

Chapter 11. Monitors

While the majority of automated traffic types are triggered by others, there is also a category that is usually under your control. This is automated monitoring software that executes regular, scheduled requests to validate aspects of system behavior.

Good or Bad?

Generally these are examples of good automated traffic, which is directly arranged by website owners in order to validate elements of the performance or availability of their systems.

However, there are some examples of other organizations that arrange monitoring traffic on websites without the consent of the website owner. There are a range of reasons why this may happen, including companies wanting to track performance of their competitors, third parties gathering stats on a range of site activity across a group of sites (e.g., industry vertical) or sales activity for companies gaining stats to use in pitches to potential customers.

Largely these external monitoring activities are harmless, at worst giving competitors some insight into your site availability and performance.

Let’s now take a look at some specific examples of monitoring traffic.

Availability

Availability monitoring usually takes the form of regular requests for a small range of pages to validate that those pages are still available, triggering alerting to site management that issues have been observed. This could be simple requests for a single page or may include a series of page requests to replicate a representative user journey. ...