Google Advertising Tools, 2nd Edition

Excluding the Bot

There are a number of reasons you might want to block bots from all, or part, of your site. For example, if your site is not complete, if you have broken links, or if you haven’t prepared your site for a search engine visit, you probably don’t want to be indexed yet. You may also want to protect parts of your site from being indexed if those parts contain sensitive information or pages that you know cannot be accurately traversed or parsed.

Note

Google requests that you block URLs that will give the bot hiccups—for example, dynamic URLs that include calendar information that have the potential for infinite expansion. You can block individual URLs using a nofollow attribute value in the anchor tag of the URL itself. For example:

<a rel="nofollow" href="botcantgohere" />No follow me</a>

Figure 4-5. Lynx Viewer makes it easy to focus on text and links without the distraction of the image-rich version

If you need to, you can make sure that part of your site does not get indexed by any search engine.

Note

Following the no-robots protocol is voluntary and based on the honor system. So all you can really be sure of is that a legitimate search engine that follows the protocol will not index the prohibited parts of your site from the root of your site (if there are external links to excluded pages, these may still be traversed regardless of your policy file). Don’t rely on search ...