O'Reilly logo

Learning AWS - Second Edition by Amit Shah, Aurobindo Sarkar

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Web Application Firewall

Serving unnecessary requests costs money. For example, blocking bad bots dynamically is a typical use case for using AWS WAF. You will need to create an IPSet containing a list of blocked IP addresses and a rule that blocks requests from these IPs. You will define a web ACL which allows requests by default and contains our rule to exclude blocked IPs. Additionally, you will need define a mechanism to detect bad bots and add their IP addresses to IPSet.

You can use robots.txt to specify which areas of your site or web app should not be scraped and to ensure there are some links pointing to non-scrapable content. Bad bots (ignoring your robots.txt) will request the hidden link and the trigger script will detect the ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required