Web Operations

Lessons Learned

We learned a lot of lessons going through this experience. We have tried to apply these lessons to everything we have done since then.

The first lesson we learned is to always plan for five years away. Ask yourself: "Can my current solution make it five years from now?" Jared Spool, a user interface expert who talks about mapping your user's experience into the future, says that five years gets you beyond the current reality but does not get you into the realm of science fiction. That is to say, you can't know what technology will exist in five years that makes your decisions different. So, plan for five years, and tomorrow will probably take care of itself.

Another lesson we learned is to test the production systems. If we had not tested our production systems, we may not have found the old ad server code that was lagging behind. This could have caused problems during our next rush, whether it was expected or unexpected. It is a scary idea, but with proper planning, it can be done.

You may notice that in our solution we did not focus on just hardware, or just database schema, or just the software. We had to rethink the whole stack. I have heard some people say that a good system will never have to be refactored. I dispute that. I think you should strive to not refactor every three years, but when it is clear that your current solution is not going to work, you have to have the courage to rethink the whole solution. We rethought our code, our hardware, our network, ...