Making Software

Generalizing from Qualitative Results

Whether you find qualitative results in a research paper or you get them yourself, one issue that always comes up is how much you can generalize from it.

For example, qualitative methods are can identify common behavior, but they cannot identify average behavior. Averages and other aggregate statistics require some ability to count, and that’s not what qualitative methods are about. Moreover, because the gathering of nonnumerical data can be more labor-intensive, qualitative methods often end up with smaller sample sizes, making it difficult to generalize to the broader population of situations under study.

Nevertheless, qualitative studies can and do generalize. For example, they can demonstrate that the cause and effect relationships present in one context are similar to those in another context. Suppose you were part of a user interface team for a web application, trying to understand why code reviews were always taking so long. Although your findings might not generalize to the team managing the database backend, they might generalize to other frontend teams for web applications in a similar domain. Knowing when the study generalizes is no less challenging than for quantitative studies; in both cases, it’s a subjective matter of knowing which assumptions still apply in new contexts.