The Ongoing Evolution of Search

Search has come a long way, and will continue to progress at a faster and faster pace. Keeping up with these changes, the competitive environment, and the impact of new technology provides a challenge and an opportunity.

The Growth of Search Complexity

Search has been evolving rapidly over the past decade. At the WSDM conference in February 2009, Google Fellow Jeff Dean provided some interesting metrics that tell part of the story:

  • Google search volume had grown 1,000 times since 1999.

  • Google has more than 1,000 times the machines it had in 1999.

  • Latency dropped from less than 1,000 ms to less than 200 ms.

  • Index update latency improved by about 10,000 times. Whereas updates took Google months in 1999, in 2009 Google was detecting and indexing changes on web pages in just a few minutes.

These are staggering changes in Google’s performance power, but this is just part of the changing search environment. Some of the early commercial search engines, such as Web Crawler, InfoSeek, and Alta Vista, launched in the mid-1990s. At that time, web search engines’ relevancy and ranking algorithms were largely based on keyword analysis. This was a simple model to execute and initially provided pretty decent results.

However, there was (and is) too much money in search for such a simple model to stand. Spammers began abusing the weakness of the keyword algorithms by stuffing their pages with keywords, and using tactics to make them invisible to protect the user experience. ...

Get The Art of SEO now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.