A robots.txt file is a text file that sits in the top-level directory of a Web site
and it tells Web crawlers or robots not to access certain pages or subdirectories of
the site.This only applies to robots that comply with Robots Exclusion Standard,
which is most search engine crawlers on the Web.The robots.txt file we located
contains the following:
#no robots
User-agent: *
Disallow: /
# Disallow Collectors and Spam
User-agent: atSpider
Disallow: /
User-agent: cherrypicker
Disallow: /
User-agent: DSurf
Disallow: /
User-agent: EliteSys Entry
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: ...