Skip to Main Content
Google Hacking for Penetration Testers
book

Google Hacking for Penetration Testers

by Johnny Long
December 2004
Intermediate to advanced content levelIntermediate to advanced
448 pages
16h 25m
English
Syngress
Content preview from Google Hacking for Penetration Testers
crawler, disallow it access to nothing.This might seem like backward logic, but the
following robots.txt file indicates that all crawlers are to be sent away except for
the crawler named Palookaville:
#Bring on Palookaville
User-Agent: *
Disallow: /
User-Agent: Palookaville
Disallow:
Notice that there is no slash after Palookaville’s disallow. (Norman Cook fans
will be delighted to notice the absence of both slashes and dots from anywhere
near Palookaville.) Saying that there’s no disallow is like saying that user agent is
allowed—sloppy and confusing, but that’s the way it is.
Google allows for extensions to the robots.txt standard. A disallow pattern ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Google Hacks

Google Hacks

Tara Calishain, Rael Dornfest
Dissecting the Hack

Dissecting the Hack

Jayson E Street

Publisher Resources

ISBN: 1931836361