Spiders, robots, and crawlers are your friends. In the name of search engine optimization, you’ll not only learn to love them, but also you’ll actually go out of your way to attract them to your site.
In SEO, spiders, robots, and crawlers are more or less synonymous, but don’t worry unduly—none have legs or feelers. So, let’s consolidate and just use the term crawler, shall we? Just bear in mind that you’ll sometimes want to attract robots, or lace your site with “spider bait”. All belong to the same principle.
So, what’s a crawler, and why would I want one on my website, anyway?
A crawler is a program or automated script (often called a ‘bot, short for robot) that scuttles ...