O'Reilly logo

The Truth About Search Engine Optimization by Rebecca Lieb

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Truth 1. Getting noticed by spiders, robots, and crawlers

Spiders, robots, and crawlers are your friends. In the name of search engine optimization, you’ll not only learn to love them, but also you’ll actually go out of your way to attract them to your site.

In SEO, spiders, robots, and crawlers are more or less synonymous, but don’t worry unduly—none have legs or feelers. So, let’s consolidate and just use the term crawler, shall we? Just bear in mind that you’ll sometimes want to attract robots, or lace your site with “spider bait”. All belong to the same principle.

So, what’s a crawler, and why would I want one on my website, anyway?

A crawler is a program or automated script (often called a ‘bot, short for robot) that scuttles ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required