Skip to Main Content
SEO Warrior
book

SEO Warrior

by John I Jerkovic
November 2009
Beginner content levelBeginner
496 pages
13h 46m
English
O'Reilly Media, Inc.
Content preview from SEO Warrior

Robots.txt

Yes, that’s right! If you are not very careful with the robots.txt file, you could be blocking web spiders from crawling content you do want to be indexed. Most reputable web spiders will obey your instructions within robots.txt. So, be very careful when you use and edit the robots.txt file. Also note that Google Webmaster Tools provides a robots.txt checker, so you can enter a URL and confirm whether a given URL will be crawled.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

The SEO Battlefield

The SEO Battlefield

Anne Ahola Ward

Publisher Resources

ISBN: 9780596804749Errata Page