SEO: Search Engine Optimization Bible

Chapter 16. Robots, Spiders, and Crawlers

IN THIS CHAPTER

• What are robots, crawlers, and spiders?

• What's the Robot Exclusion Standard?

• Robots meta tag

• Inclusion with XML site mapping

You're nearing the end of the book, so you should have a pretty good handle on what exactly robots, spiders, and crawlers are, right? No doubt you do, but did you know that there is much more to these Internet creatures than just the fact that they crawl from one web site to another?

Spiders, robots, crawlers, or whatever else you choose to call them can determine how well you rank in search engines, so it's best to make friends with them as quickly as possible. Certain strategies will help you find favor with these crawlers (which is the name we'll use to lump them all together), and others, unfortunately, will help you find your way right out of search engine rankings.

A sad fact of crawlers is that sometimes you'll be treated unfairly when you've done absolutely nothing wrong. Some of the strategies that you may want to use to improve the optimization of your site can look suspicious to crawlers. They'll see those strategies and automatically penalize you for trying to spam the search engine, even when you're not.

That's what makes it so important for you to understand not only the intimate details about how crawlers work, but also about what makes them happy and what makes them move on without looking back.

What Are Robots, Spiders, and Crawlers?

You should already have a general understanding that a robot, ...