O'Reilly logo

Managing Startups: Best Blog Posts by Thomas Eisenmann

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 24. Introduction to A/B Testing for Landing Pages

Everyone loves the story about how a scrappy upstart changed the color of a button on its landing page and suddenly increased its conversion rate by 200%. Stories like that make for great headlines. They also often spawn copycats naïvely trying to replicate that anomaly in completely different contexts.

Do you remember when Dustin Curtis ran an experiment trying different ways of wording a link to his Twitter account? He noted that writing “You should follow me on Twitter” had a greater conversion rate than just “I’m on Twitter.” In response, blogs and websites all over the world began demanding I follow them on Twitter in increasingly imperative tones. Did that trick actually increase the Twitter followings of anyone? I have no idea, and neither do the majority of the people who tried it. It really seems like a “tip” that you can easily port to another context and have similar results. However, there is no way to predict what results any change will have—either positively or negatively.

Incremental changes that lead to huge jumps in performance are rare. To be more specific, incremental changes that lead to statistically valid jumps in performance are rare. Most websites do not get anywhere near enough traffic for the data collected in such an experiment to mean anything. It is easy to claim that no two snowflakes are identical when you’ve only looked at a few. It will take a couple of blizzards before you can ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required