Perl for Web Site Management

Chapter 18. Monitoring Search Engine Positioning

The first commercial web site I worked on full time taught me many useful lessons. One of those lessons came about 18 months after the site’s unveiling, when it suffered a sudden unexplained drop in visitor traffic. After a lot of research and a bit of hair-pulling, I finally figured out what had happened: I had invalidated a large number of offsite search engine links by changing the location of some pages on the site. The search engines eventually noticed all the resulting 404 errors at the destinations to which they were referring searchers and removed those pages from their databases. It took months to reestablish the site’s “footprint” in the search engine databases and rebuild traffic to previous levels.

Needless to say, this brought home to me the importance of maintaining stable URLs for web resources. In a very real sense, your web site does not exist in isolation. It is simply a small component in a much larger, distributed system, and moving resources from one location to another breaks that larger system. It doesn’t matter that you can update all the links on your site so that they work correctly with the new locations; offsite links will still be broken. The only really good way to avoid that is to make sure you maintain stable URLs for your web resources forever (or at least for as long as is practical).

The other thing that this incident brought home to me was the importance of third-party search engines as a source ...