Web Operations

Monitoring and History of Patterns

In the middle of the night, late 2008, we were migrating a huge number of websites from one GSLB platform to another. We'd done plenty of due diligence ahead of time, to ensure that the change in proximity routing algorithms wouldn't significantly affect the traffic flows. Mid-migration, the owners of one site suddenly panicked; they saw a significant spike in traffic on their East Coast site! Was the new GSLB configuration wrong, or was something broken? We spent more than 30 minutes debugging the problem, but everything we looked at told us the platform was functioning correctly. Eventually, somebody thought to look at a weeklong traffic pattern history, instead of just the last few hours most of us were monitoring. As it turned out, that site had a traffic spike of identical size on the East Coast every day at the same time. It was just a search engine doing its daily site scrape, not at all related to the migration. We'd blown 30 minutes of many engineers' sleep time chasing nothing but normal. If the site owner had been aware of his normal traffic patterns, it would have saved everyone much grief.

The moral of this story is to know your daily, weekly, and monthly patterns. If you are aware of the oddities of your normal traffic flows, you won't be surprised during failovers, migrations, or upgrades. Make sure your monitoring includes week-on-week graphs and trending, which includes not just aggregate traffic levels but also what goes where ...