Hands-On Penetration Testing with Kali NetHunter

Using Parsero

In terms of Kali, there actually are tools that will look for the robots.txt of a site and read the contents. One such tool, Parsero, is specifically designed to look at the contents of the file with attention paid to the disallow entries that are used to keep bots from visiting those locations. The following screenshot shows Parsero in action:

The Parsero welcome screen

Parsero reads the robots.txt file of a web server, looks at the Disallow entries, and then checks to see whether the location is accessible. In practice, an entry in the file would look like this Disallow: /portal/ login means that the content on this website ...