Search Engine Optimization Bible, Second Edition

Chapter 18. Robots, Spiders, and Crawlers

IN THIS CHAPTER

• What are robots, spiders, and crawlers?

• What's the Robot Exclusion Standard?

• Robots meta tag

• Inclusion with XML site mapping

You're getting well along in the book, so you should have a pretty good handle on what exactly robots, spiders, and crawlers are, right? No doubt you do, but did you know that there is much more to these Internet creatures than just the fact that they crawl from one web site to another?

Spiders, robots, crawlers, or whatever else you choose to call them can determine how well you rank in search engines, so it's best to make friends with them as quickly as possible. Certain strategies will help you find favor with these crawlers (which is the name we'll use to lump them all together), whereas other strategies, unfortunately, will help you find your way right out of search engine rankings.

A sad fact about crawlers is that sometimes you'll be treated unfairly when you've done absolutely nothing wrong. Some of the strategies that you may want to use to improve the optimization of your site can look suspicious to crawlers. When they see those strategies they will automatically penalize you for trying to spam the search engine, even when you're not.

That's what makes it so important for you to understand not only the intimate details about how crawlers work, but also about what makes them happy and what makes them move on without looking back.

What Are Robots, Spiders, and Crawlers?

You should already have a general ...