Making Software

What Are Qualitative Methods?

Put simply, qualitative methods entail the systematic gathering and interpretation of nonnumerical data (including words, pictures, etc.). Like quantitative methods, qualitative methods can be used to gather data to confirm or reject beliefs (deductive reasoning). However, qualitative methods can also be used to support inductive reasoning: gather data to arrive at new explanations. Because qualitative methods gather nonnumerical data, they also lend themselves to being used in more natural settings.

To illustrate, let’s focus on a specific scenario and discuss how qualitative methods might be used to understand it. Imagine you’ve just started managing a software team. You’re leading a new project and have just adopted a new bug tracking system with some great features your team has been dying to have. Over the next few months, you see your team patching a lot of code and making great progress, but every time you check the bug list in the new system, there are only a few reports from the same few testers and they never seem to change. Then, one morning you walk by your best developer’s desk and see the old bug tracker up on his screen. Your team has been secretly using the old system for weeks! After all of that training, the careful transition, and the costly licensing, why aren’t they using the new tracker?

One obvious thing to do is simply ask. For example, you could call a meeting and just ask the team to explain why they’re avoiding the new system. You’d get some explanation, but it may not be entirely trustworthy, especially if the developers find themselves opposing your decision to adopt the new tracker. Moreover, the fact that they’re in a social context will also make the less talkative members of your team less likely to speak, biasing the explanations you get to your most vocal employees. A biased, warped explanation won’t be helpful to anyone.

To take yourself out of the equation, perhaps you could ask a trustworthy friend to ask around during coffee breaks, conducting brief, informal interviews. That would give each person a chance to state his opinion outside of a group context, perhaps freeing him to be more vocal and more honest. But you’ll still be limited to the views of your friend’s friends. Even worse, your friend might not be impartial; maybe he particularly dislikes the new tracker and unintentionally biases his report back to you.

The issue with the approaches I’ve just suggested is that they are secondhand or even thirdhand accounts. Ideally, you would be able to observe the moment of interest. For example, when people on the team decided to use the old tracker instead of the new one, what was going on in their heads? What were the other constraints on their time? Who were they collaborating with? What data were they entering? To get at these questions, you might sit next to developers for a day, watching what they do. This would allow you to directly observe the moments when they decide to use the old tracker instead of the new one.

Of course, directly observing people in these moments is often unfeasible. People don’t like being watched, and often adapt their behavior to preserve their privacy or avoid embarrassment or anxiety. Moreover, since you’d be the sole observer, you’re likely to bring biases to your observations.

It might also be possible to take people out of the picture altogether and just study the documents. Which bugs are getting reported in the old system, and which are reported in the new system? This might uncover differences in the type of reports that the team hasn’t been filing in the new system. This might give you a hunch about the reasons for using the old tracker, but it still wouldn’t tell you the actual reasons inside the minds of your team.

All of these approaches are qualitative methods and all of them have limitations. The solution to getting around these limitations is to embrace them: no one approach will reveal the whole unbiased truth. Instead, good qualitative research combines multiple methods, allowing one to triangulate evidence from multiple sources. For example, suppose you interviewed individual employees about the tracker but also studied which bugs were being filed in the new tracker. Each approach will give you a distinct story, more or less consistent with the stories from other approaches. By comparing and contrasting these stories, one can uncover the ground truth behind a question.

Aside from gaining understanding, qualitative methods are also good for gaining empathy. More often than not, the cause of communication breakdowns, broken processes, and user frustration is people’s inability to see the world from another person’s perspective. And this is precisely what qualitative methods are designed to correct. This perspective-taking is often precisely the goal of qualitative methods. For example, suppose you manage a team and have noticed that every Friday the build breaks, people spend the whole day trying to fix it, and everyone goes home frustrated. Using a qualitative approach to diagnose the cause of these broken builds might lead you to discover that Thursday nights are a big night for check-ins because you’ve decided Friday is meeting day. Knowledge like this can help you see the world from the developers’ perspective and find solutions that make everyone happy.