Truth 1. Getting noticed by spiders, robots, and crawlers
Spiders, robots, and crawlers are your friends. In the name of search engine optimization, you’ll not only learn to love them, but also you’ll actually go out of your way to attract them to your site.
In SEO, spiders, robots, and crawlers are more or less synonymous, but don’t worry unduly—none have legs or feelers. So, let’s consolidate and just use the term crawler, shall we? Just bear in mind that you’ll sometimes want to attract robots, or lace your site with “spider bait”. All belong to the same principle.
So, what’s a crawler, and why would I want one on my website, anyway?
A crawler is a program or automated script (often called a ‘bot, short for robot) that scuttles ...
Get The Truth About Search Engine Optimization now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
