Skip to Main Content
Search Engine Optimization Bible, Second Edition
book

Search Engine Optimization Bible, Second Edition

by Jerri L. Ledford
April 2009
Beginner to intermediate content levelBeginner to intermediate
528 pages
14h 12m
English
Wiley
Content preview from Search Engine Optimization Bible, Second Edition

Chapter 18. Robots, Spiders, and Crawlers

IN THIS CHAPTER

  • What are robots, spiders, and crawlers?

  • What's the Robot Exclusion Standard?

  • Robots meta tag

  • Inclusion with XML site mapping

You're getting well along in the book, so you should have a pretty good handle on what exactly robots, spiders, and crawlers are, right? No doubt you do, but did you know that there is much more to these Internet creatures than just the fact that they crawl from one web site to another?

Spiders, robots, crawlers, or whatever else you choose to call them can determine how well you rank in search engines, so it's best to make friends with them as quickly as possible. Certain strategies will help you find favor with these crawlers (which is the name we'll use to lump them all together), whereas other strategies, unfortunately, will help you find your way right out of search engine rankings.

A sad fact about crawlers is that sometimes you'll be treated unfairly when you've done absolutely nothing wrong. Some of the strategies that you may want to use to improve the optimization of your site can look suspicious to crawlers. When they see those strategies they will automatically penalize you for trying to spam the search engine, even when you're not.

That's what makes it so important for you to understand not only the intimate details about how crawlers work, but also about what makes them happy and what makes them move on without looking back.

What Are Robots, Spiders, and Crawlers?

You should already have a general ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Search Engine Optimization All-In-One For Dummies®

Search Engine Optimization All-In-One For Dummies®

Bruce Clay, Susan Esparza

Publisher Resources

ISBN: 9780470452646Purchase book