Practical A/B Testing

Understanding the Cost of Long-Term Holdbacks

We spend an estimated third of our lives sleeping. So buying an expensive mattress or overpriced flannel bed sheets may initially seem like a waste. However, once the cost is correlated to how much time is spent sleeping, it’s pennies in comparison. Similar to how an expensive mattress can improve one’s sleep dramatically, long-term experiments may seem expensive initially but are very much worth it in the long run.

As powerful as long-term experiments are, let’s recognize the big cost elephant in the room. Maintaining multiple versions of a product or feature, especially on a longer timeline, has a cost. It’s easy to make mistakes when managing multiple versions of codebases or maintaining a holdback ...