Facebook Cookbook

Continuous Improvement Through A/B Testing

Problem

I want to continuously improve my ability to attract new users, but I don’t know how to keep evaluating what’s working.

Solution

A/B testing comes from the world of hi-fidelity audio systems, in which people stand in neutral rooms and toggle back and forth between the “A” and “B” speakers connected to an amplifier, to decide whether they should buy the ones that cost twice as much as your mortgage or the ones that are more expensive than a university education. Luckily for you, A/B testing isn’t just for audiophiles anymore! The same principle can be applied to web or Facebook apps: develop “A” and “B” alternatives, show “A” to a test group and “B” to another test group, and see which one is better at a predefined metric. You might, for example, try different invite texts to see which attracts users faster or at a higher response rate.

Discussion

Implementing your own A/B test framework is doable, but that’s like saying it’s possible to climb Mount Everest. If you’re just getting started, don’t invest a huge amount of time in building something that isn’t core to your application, because it just drags you away from the important stuff. Take a look at efforts such as The Fountain Project (http://www.fountainproject.com/), which provides an A/B system for testing Facebook invitations.