Web Operations

How Do We Know It Works?

We did all this work over the course of several months. But how did we know we made any improvements? Although we ran simple benchmarks along the way to see how things were working, that is no substitute for a real test.

Luckily for us, our COO was previously with Keynote Systems. Keynote is a company that (among other things) can perform load tests on websites. In our opinion, the only way to make sure our production systems could handle the load was to test them, live, during the day when traffic was flowing. When I tell people this, they look shocked. But how would we know if it worked otherwise?

We scheduled a test. We notified our content team when the test was going to occur, and we provided test scenarios to Keynote that would accurately simulate the distribution of different types of requests. Keynote's systems could hit our website from several different datacenters with multiple clients per datacenter. The test scenario could also slowly ramp up the number of requests over time. This way, if things started to go bad, we could cancel the test with the push of a button.

The first test we ran did not tell us much. We had it configured to test the pages and all objects on them. The problem was that all it did was load-test our CDN provider. Because one page request to us would then generate 10–200 requests to our CDN, our servers never felt any load.

Next, we had the test ignore any objects that were on our CDN. This would focus the testing nodes on our ...