CHAPTER 12
robots.txt
In your Global Configuration settings, and in the articles and menus, you have often seen, and possibly used, the Robots field. This is used to indicate whether certain pages of your web site should be indexed or not.
While not entirely the same, the reason for the existence of the robots.txt file (Listing 12-1) in the root of your web site is somewhat similar. You can use it to inform spiders and robots from search engines which part of your web site’s folders should be opened for spidering. When you open it up in an editor, you will see the following, except for some commented out lines:
Get Joomla! 3 SEO and Performance now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
