INSPIRED, 2nd Edition

CHAPTER 54Quantitative Value Testing Techniques

While qualitative testing is all about fast learning and big insights, quantitative techniques are all about collecting evidence.

We will sometimes collect enough data that we have statistically significant results (especially with consumer services with a lot of daily traffic), and other times we'll set the bar lower and just collect actual usage data that we consider useful evidence—along with other factors—to make an informed decision.

This is the main purpose of the live‐data prototype we discussed earlier. As a reminder, a live‐data prototype is one of the forms of prototype created in product discovery intended to expose certain use cases to a limited group of users to collect some actual usage data.

We have a few key ways to collect this data, and the technique we select depends on the amount of traffic we have, the amount of time we have, and our tolerance for risk.

While qualitative testing is all about fast learning and big insights, quantitative techniques are all about collecting evidence.

In a true startup environment, we don't have much traffic and we also don't have much time, but we're usually fine with risk (we don't have much to lose yet).

In a more established company, we often have a lot of traffic, we have some amount of time (mostly we're worried about management losing patience), and the company is usually more averse to risk.

A/B Testing

The gold standard for this type of testing is an A/B test. The ...