Yahoo! Hacks

Hide Part of Your Web Site from Yahoo!

Though many sites do whatever they can to be found and ranked highly in Yahoo! Search results, there might be parts of your site that you want to keep private.

The Web is considered a public place, used to share pages of information with anyone in the world who wants to view them. But, to a lesser degree, the Web is also used to share information with small groups, or even a single individual. Because Yahoo! indexes as much of the Web as it can, these semipublic spaces can also be included in Yahoo! Search results. With just a bit of work, you can tell Yahoo! exactly which pages are meant for public consumption and which pages shouldn’t be included in Yahoo! Search results. In addition, some sections of sites might not be a good introduction to the site, and you might want to control where people enter.

Yahoo! scans the Web with a program called Slurp. Slurp is a bot (short for robot) that visits and indexes web pages, makes a copy of the page for Yahoo!, and follows any links in the page looking for more pages to index. In addition to standard web pages, Slurp copies other files it finds along the way, such as PowerPoint presentations, PDF files, Word documents, Excel spreadsheets, and XML data files. Because of Slurp’s link-following nature, many people think that if a page or document isn’t linked from a page on their site that Yahoo! won’t find it and it’ll be out of view. But Slurp doesn’t follow links exclusively; Slurp also looks for ...