Managing and Mitigating Bots

Chapter 1. What Is Automated Traffic?

There is a range of different definitions of what can be classed as automated traffic.

For example, Frost & Sullivan describe bot traffic as “computer programs that are used to perform specific actions in an automated fashion,” Akamai has defined it as “automated software programs that interact with websites,” and Wikipedia defines a bot as “a software application that runs automated tasks (scripts) over the Internet,” whereas Hubspot says “A bot is a type of automated technology that’s programmed to execute certain tasks without human intervention.”

For the purposes of this book I will use the following description for automated traffic, which I feel captures the essential details of what is meant by the term and removes some of the vagaries included in the other descriptions:

Automated traffic is any set of legitimate requests made to a website that is made by an automated process rather than triggered by a direct human action.

History of Automated Traffic

The history of the type of automated traffic I am discussing here can be traced back to 1988 with the creation of IRC bots such as the Hunt the Wumps game and Bill Wisner’s Bartender. It wasn’t until 1994, however, that the first search engine spiders were created by WebCrawler (later purchased by AOL). GoogleBot followed in 1996.

Key Characteristics of Automated Traffic

For the purposes of this book, I will have a limited definition of automated traffic; this is not to say that ...