The Art of SEO, 2nd Edition

Chapter 2. Search Engine Basics

In this chapter, we will begin to explore how search engines work. Building a strong foundation on this topic is essential to understanding the SEO practitioner’s craft.

As we discussed in Chapter 1, people have become accustomed to receiving nearly instantaneous answers from search engines after they have submitted a search query. In Chapter 1 we also discussed the volume of queries (more than 6,000 per second), and Google reported as early as 2008 that it knew of about 1 trillion pages on the Web (http://googleblog.blogspot.com/2008/07/we-knew-web-was-big.html). It is likely that this number is now low by one or more orders of magnitude, as the Web continues to grow quite rapidly.

Underlying this enormous data processing task is the complex nature of the task itself. One of the most important things to understand about search engines is that the crawlers (or “spiders”) used to visit all the web pages across the Web are software programs. Software programs are only as smart as the algorithms used in implementing them, and although artificial intelligence is being increasingly used in those algorithms, web crawling programs still don’t have the adaptive intelligence of human beings.

Software programs cannot adequately interpret each of the various types of data that humans can—videos and images, for example, are to a certain extent less readable by a search engine crawler than they are through the eyes of humans. These are not their only limitations, ...