Managing and Mitigating Bots

Introduction

Web traffic consists of more than just the human users who visit your site. In fact, recent reports show that human users are becoming a minority. The rest belongs to an ever-expanding group of traffic that can be grouped under the heading automated traffic.

Terminology

The terms automated traffic, bot traffic, and non-human traffic are equally common and are used interchangeably throughout this book.

As long ago as 2014, Incapsula estimated that human traffic only accounted for as little as 39.5% of all traffic they saw. This trend is predicted to continue, with Cisco estimating that automated traffic will grow by 37% year on year until 2022.

However, this is not simply a growth in the quantity of automated traffic but also in the variety and sophistication of that traffic. New paradigms for interaction with the internet, more complex business models and interdependence between sources of data, evolution of shopping methods and habits, increased sophistication of criminal activity and availability of cloud-based computing capacity, are all converging to create an automated traffic environment that is ever more challenging for a website owner to control.

It’s Not All Good or Bad

It is simplistic to think of automated traffic as being all goodies and baddies. However, the truth is much more nuanced than that. As we’ll discuss, there are clear areas of good and bad traffic but there is a gray area in between where you will need to assess the positivity or negativity ...