November 2009
Beginner
496 pages
13h 46m
English
Yes, that’s right! If you are not very careful with the robots.txt file, you could be blocking web spiders from crawling content you do want to be indexed. Most reputable web spiders will obey your instructions within robots.txt. So, be very careful when you use and edit the robots.txt file. Also note that Google Webmaster Tools provides a robots.txt checker, so you can enter a URL and confirm whether a given URL will be crawled.