• Friday, November 28
By midday Thursday, right in the middle of Thanksgiving, we knew we were in trouble. The overnight Unicorn e-mail promotion was an incredible success. The response rate was unprecedentedly high, with traffic to our website surging to record levels, which kept bringing down our e-commerce systems.
We initiated an emergency Sev 1 call, putting in all sorts of emergency measures to maintain our ability to take orders, including putting more servers into rotation and turning off computationally-intensive features.
Ironically, one of the developers suggested turning off all the real-time recommendations, which we had worked so hard to build. Why recommend more products to buy, he argued, if customers can’t even complete ...