Web Security Testing Cookbook

Chapter 6. Automated Bulk Scanning

For many years it was believed that countless monkeys working on countless typewriters would eventually reproduce the genius of Shakespeare. Now, thanks to the World Wide Web, we know this to be false.

—Robert Wilensky

Automation is a tester’s friend. It gives you repeatability, consistency, and better coverage over the software. From a security point of view, you have so much to test that you have to automate in order to have any confidence that you’re covering enough interesting security test cases.

In Chapter 1, we talked about how vital it is to narrow our focus and to get a manageable number of security tests. Even after narrowing our focus, we’ve got a small slice of infinity to test. That’s where automation comes in. This chapter gives you some tools that can help you automate by programmatically exploring your web application. There are two kinds of tools we’ll discuss: those that systematically map a website and those that try to automatically find security problems.

Mapping tools are typically called “spiders” and they come in a variety of shapes and sizes. They fetch a starting page that you tell them to fetch, and then they parse that web page. They look for every link on the page and then they follow it. After following the link, they read that page and record all the links from it, and so on. Their goal is to visit every web page in your application.

There are a few benefits to mapping your website with a tool like this. You get an inventory ...