In your Global Configuration settings, and in the articles and menus, you have often seen, and possibly used, the Robots field. This is used to indicate whether certain pages of your web site should be indexed or not.
While not entirely the same, the reason for the existence of the robots.txt file (Listing 12-1) in the root of your web site is somewhat similar. You can use it to inform spiders and robots from search engines which part of your web site’s folders should be opened for spidering. When you open it up in an editor, you will see the following, except for some commented out lines:
Listing 12-1. robots.txt File ...