Chapter 4. Plans Make Research Work

In the Introduction, we talked about excuses for not doing research. Often, the people who make those excuses have experienced failed research efforts. Planning your research is key to ensuring success. But because planning is often invisible from the outside, if you are new to product research, you may have taken it for granted.

What if ready, aim, fire is better than fire?

This chapter is about planning, including picking a method, finding participants, working as pairs, preparing guides for your sessions, and keeping all related parties in the loop—as well as what to do when things go wrong.

Picking a Research Method

If a sibling or friend has symptoms of the flu, how would you find out how bad they feel? You’d ask them. You wouldn’t run them through a CT scan to find out how they feel. On the other hand, if you suspected they might have a lung infection, it would be more appropriate to run some tests than to simply ask them how likely they feel it is that they have a lung infection.

How you should go about conducting research depends on what you want to know. You need to pick the right research method for your research question. Running a CT scan to understand how someone feels is like running a usability study to understand how much future customers would pay for your product. And asking someone to judge if they have a lung infection based on how they feel is like sending out a survey to understand why people abandon their carts online. Different research questions require different methods—and, just like in medicine, if you pick the wrong process, you might end up missing something vital.

Product research combines user research and market research with product analytics to gain real insights into how we should design and improve products. Each of these three research disciplines has multiple subcategories (see the Introduction). Choosing which one to employ will depend on your research question: what do you want to know and why?

There is a wide array of philosophies and frameworks on research methods and their use in product development.1 These frameworks organize research methods into different groups so that you can determine which method to use. We have a simple grouping based on two questions. Ask yourself these two questions to determine which research methods you should use:

Question 1:
Which stage of product development are you in?
Question 2:
To answer your research question, do you need to understand attitudes and behaviors at a personal level, needs and motivations at scale, or detailed usage patterns over time?

Table 4-1 (opposite) organizes types of research according to these questions. (If you need a refresher on the stages of product development and types of research, feel free to revisit the Introduction.)

Table 4-1 will help you pick methods that are compatible with your research question. For example, if you are about to release a product and you want to know about what marketing messages will resonate with your target users, it will direct you to a subset of market research methods. It will prevent you from using generative user research methods, for example.

Table 4-1. Research methods organized by stages of product development and the nature of your research question
Which stage are you in? What do you need to understand? Suggested approaches Suggested methods
Stage 1 Attitudes and behavior Generative user research Ethnographic studies, contextual interviews, participatory design
Descriptive user research Interviews, contextual interviews, diary studies, user session video playbacks
Needs and motivations Descriptive market research Interviews, surveys
Stage 2 Attitudes and behavior Descriptive user research Interviews, contextual interviews, diary studies, user session video playbacks
Evaluative user research Usability studies, multivariate (A/B) testing, surveys, eye-tracking
Needs and motivations Exploratory market research Secondary/desktop search, benchmarking, interviews, competitive tracking
Descriptive market research Interviews, surveys
Predictive market research Conjoint analysis
Usage patterns Diagnostic analytics Data drilldowns, correlation and causation
Stage 3 Attitudes and behavior Evaluative user research Usability studies, multivariate (A/B) testing, surveys, eye-tracking
Needs and motivations Exploratory market research Secondary/desktop search, benchmarking, interviews, competitive tracking
Descriptive market research Interviews, surveys
Causal market research Multivariate (A/B) tests, field trials
Predictive market research Conjoint analysis
Usage patterns Descriptive analytics Cohort analysis, segmentation, funnel or clickstream analysis, pirate metrics (AARRR)a
Diagnostic analytics Data drilldowns, correlation and causation
Predictive/prescriptive analytics Regression modeling, machine learning, correlation/causation experimentation

a Dave McClure, “Startup Metrics for Pirates: AARRR!,” Master of 500 Hats (September 6, 2007), https://500hats.typepad.com/500blogs/2007/09/startup-metrics.html.

While Table 4-1 is useful in giving you a small number of methods to pick from, it doesn’t give you the exact method to use for your research. No single research framework can achieve this, as there are so many variables that need to be considered to model your exact, unique circumstances.

That being said, there are a few other parameters you can look at to further reduce the number of methods you should use.

Required Skills Versus Available Skills

Do you have the skills required to apply the methods you see? If not, how hard will it be to find someone who can? What will the cost be, and when will they be available? Gravitate toward the methods that you can apply yourself or with external help that is available soon.

Cost of the Method

What is the cost of applying each method? This can be monetary cost or opportunity cost. For example, many teams are excited about running usability studies, but they underestimate the effort required to run a usability study with a prototype. Building a prototype that you can give the participants to complete tasks without you helping them is not an easy thing to do. The prototype needs to work for the major user flows, and that is not always a simple thing to do. If an interview where you show the participants static screens to get their feedback would suffice, there is no need to spend days or weeks adding edge cases to a prototype and polishing it to look slick. Choose methods that have a lower cost to you.

Cost of Recruitment

Different methods may require different types of participants. This depends on the nature of the method and what it requires from the participant. There may be cases where you have two methods and recruiting for one of those methods is far easier than recruiting for the other. Consider ease of recruitment as a factor in picking a method. But be careful not to make this your primary decision driver, as it will lead you to wrong conclusions. More on this in the next section.

Picking the right method ensures that you are spending your efforts wisely, in a way that will have returns. A wrong, mismatched method will produce weak insights at a high cost, whereas a method that is capable of answering your question will get you insights efficiently.

Try It Out: Pick a Method

In Table 4-2, you will see a list of research questions from a hypothetical product team and the stages at which they might pose these questions. We’ve picked just a few methods for this exercise. Try to identify what they want to learn in each step, and think about the methods they could use. You can cover the right side of the table and work through each question.

Table 4-2. Research questions at each stage of the research process
Research question and stage asked Do they want to learn about attitudes and behavior, needs and motivations, or usage patterns? Example methods
What are the emotional aspects of saving money? (Stage 1) Attitudes and behavior; maybe also needs and motivations This is a great generative user research question or an exploratory market research question. Interviews or diary studies are two good methods to use to explore this question.
What paths in the app result in reduced conversion rate? (Stage 3) Usage patterns This question requires a detailed analysis of how users move through the app, where a diagnostic analytics approach may work. This is not predictive analytics as they are not making any future inferences.
What drives demand to our product in the Asia-Pacific region? (Stage 3) Needs and motivations This is a broad question that can be answered through exploratory and descriptive market research. Desktop research, benchmarking, and surveys would be great methods.
How does the new category hierarchy influence users’ navigation choices? (Stage 2) Attitudes and behavior As the team is looking for particular behaviors caused by the new navigation, evaluative user research methods like usability studies would be appropriate.

Finding Participants

Imagine you’re building an aircraft maintenance system for airport workers. Would you test the prototype with a group of florists? While that could be a fun study to watch, you wouldn’t learn much from, it and, aside from some possibly GIF-worthy slapstick, you’d be wasting everyone’s time. Yet this is what we do when we ask random people on the street to test a product: we seek information from people whose needs may be totally unrelated to those of our users. So how do you ensure you’re researching with the right users for your product?

Identifying your participants is the first step to targeted product research. Finding the right participants for your study isn’t as simple as walking into the cafeteria. Recruiting the right people to work with and narrowing them down to those particularly suited to the study are crucial.

The Easy-to-Reach Audience Trap

We want to remind you about availability bias, which we discussed in Chapter 2. Unfortunately, it is too easy to gravitate toward the most available group of people for your research project, but that does not yield good results. Aras witnessed this when he worked on a project along with a famous, expensive design firm; let’s call them the Really Big Agency (RBA). RBA was very good at creating great-looking interface designs, but they dragged their feet about getting customer feedback in the design cycle. Aras’s team pressured RBA to do a quick usability study. RBA’s team then took the prototype to their colleagues and families. They came back and reported that “the app is fine.” Aras’s team was not convinced, so they set up their own (properly planned) usability study. The results were abysmal. None of the 15 participants used the beautifully crafted dashboard as RBA had envisioned.

Colleagues and family members may tell you what you want to hear, but they are probably not exhibiting the behaviors and attitudes you are trying to learn about in your research. Their answers are likely to cost you a lot of time and money in poor product development. Conversely, recruiting the right participants and ensuring that they represent the users you’re trying to reach will save time, money, and perhaps embarrassment in the long run.

Considerate Selection: The Screening Process

A screener is a simple questionnaire that allows you to check whether someone can provide relevant input into your research. Good screeners are short and easy to answer, usually made up of a few multiple-choice questions.

It is tempting to ask for demographic data in the screeners, such as age, gender, and occupation. We recommend asking for demographics only if they are directly related to your research question or if you are aiming for a certain distribution among your participants. Instead, ask about behavior and attitudes. Recent activity and explicitly stated opinions are quicker and more reliable ways to figure out if these are the right participants to work with.

An easy way to come up with screeners is to base them on existing user groups. Segments and cohorts (covered in Chapter 3) could be a good place to start. For our athlete-coaching app, an example of this might be “coaches who already use a coaching app.” Your study can target more than one group. Once you know who you want to work with, compose a set of questions to find them based on objective behaviors, not demographics.

One of the richest sources for recruiting participants might be right under your nose: the data you already have. You probably analyzed some of it to help identify your research question. You may already have information about how your customers use your product, what their history is, and possibly simple demographics. These data points give you the same kind of historic behavior data that screeners do.

Teams who are new to research sometimes feel they need to talk to hundreds and hundreds of people to get solid insights. That is partially true. The number of users you need to recruit depends on the method you are using, which is determined by your research question. If you are exploring the needs of refugees displaced into another country by local conflict, you can gather amazing insights by talking to just a few of them (probably through interviews). On the other hand, if you are trying to see if your new design brings in more conversions, you will need to reach thousands or maybe millions of users (probably through A/B testing).

Insights that inspire others and invite them to action can come from any number of users. If one user among your carefully selected set shows you a genuine use of your product that challenges some assumptions, you can’t say, “That isn’t real because other people don’t exhibit that behavior.”

Keeping Track of Participants

Screening participants by collecting data from them in advance makes it possible for you to get to the right people for your research needs. However, the data that you collect may be very sensitive. Therefore, it is very important to take measures to maintain the privacy of your participants.

The database where you keep your participants’ information and screener responses is likely to include private information like emails, home addresses, and phone numbers. Depending on your research needs, you may also have sensitive personal information, such as participants’ sexual orientations, religious beliefs, political stances, and incomes. Full access to these records should be limited to only a few people. Give each participant a short, unique identifier, such as a number or alphanumeric code. You’ll use this when taking notes or during analysis. The person who has access to the file should be responsible for filtering the full set of participants available based on the needs of each research project. They should only share with their colleagues the subset of the data that pertains to suitable participants for the current research project.

When a participant takes part in research, you should make a note of that in your database. This lets you make sure that you don’t work with the same participant over and over again, especially over short periods of time. Why? Bias! Participants who are consulted too frequently might assume that you are talking to them because they are soooo great at using your product, that you came back to them because they performed really well in the previous studies, or that they have great ideas that you love! (We talked about biased participants and social desirability in Chapter 2.) Determining how long you should wait until you call a participant for another study will help you control these biases. How long you should wait depends on your product and research, but most teams wait about six months to go back to a participant.

Keeping track of participation also helps you weed out incentive hunters. There is, unfortunately, a group of people who enroll in research programs just to receive the research incentive. They are very good at providing fake answers to screeners so that they appear to qualify for a study. Not only do they provide poor feedback, but they also take the place of someone else who could provide good feedback. When you encounter an incentive hunter, make a note in the participant database. Some may even try to game your checks by using different names and email addresses. If you suspect that someone is not a genuine participant, you can contact them in advance and screen them in person.

Recruiting a new set of users for each research project can be time-consuming and expensive. Creating a group of users you can come back to time and again will make multiple studies with short sprint times a lot more efficient. Such groups of users are called research panels. Although they’re fairly labor intensive to set up, research panels shorten the time to get to relevant users and make it very easy for everyone in the company to do research. In Chapter 9, you’ll see how fashion ecommerce company Zalando does this.

Finding an Emotional Incentive

When you are screening for interviews or usability studies, there is a special group of people you should seek out: people who are naturally inclined to share their opinions and experiences with you.

What’s in it for your participant? Is it just a $20 gift card? Think about what you offer as an incentive to be part of the study. Gift cards are useful, but you’ll get more engaged participants if you can reach people who are driven to provide their input. Note that they may be driven to provide both positive or negative feedback—be open to both.

When C. Todd was testing some prototypes with manufacturers at MachineMetrics, he knew that the testing was not just a step along the way—he was solving someone’s problem. So rather than offer a $20 gift card, he offered the participants an opportunity to shape the product to meet their needs. C. Todd has lost count of how many people he’s talked to over the years who have asked for the $20 gift card up front; their participation in the study was more about an economic transaction than an exchange of quality information. Find customers who have a distinct problem you can solve so that the benefit to them isn’t just a free lunch.

The downside of this approach is its potential for selection bias. You can avoid this by expanding your screener with questions about the participant’s attitude and recent problems and moments of joy they had with your product.

To simplify the screening process, try our next approach: going to where your participants are.

Going Where Your Users Are

One team C. Todd spoke with a few years ago was developing an app aimed at athletic coaches (not the Beachbody company previously mentioned). In digging deeper, the team revealed that the app was more specifically aimed at coaches who worked with endurance athletes. Without this distinction, it would have been harder for that team to reach the specific users they were targeting. Narrowing your users down to the right niche will ensure that you’re working with the people who’ll actually use your product.

In this case, the app developers knew they wouldn’t find their target users at their local Starbucks or even by posting an ad on Craigslist. So where did endurance coaches hang out? Was there a secret endurance-coach club they could visit? No, but there were running and cycling races, as well as niche websites and forums where these coaches spent time. The product team was able to get to the right participants by going where they were, physically and virtually.

Going to where your users are helps with two things. First, it brings you to actual users of your product and makes participant selection easier. Second, you get to witness where and how people actually use your product. (We will talk about a research method that is built on this principle in Chapter 6.)

How can you make sure that you are talking to the right participants if you haven’t screened them ahead of time? You can prepare a short screener and present it verbally when you first engage with participants. If they are a good match, you can proceed with your research. If they are not, you can see if they qualify as a valuable extreme user. If that is not the case either, you can thank them and continue with a new participant.

Atlassian, the maker of Jira, uses this technique to gather relevant feedback from users. Its researchers set up booths at IT conferences where Jira users go, to draw in and meet with passionate users who want to provide feedback. They create fun and engaging spaces to interact with conference goers in short bursts during breaks (Figure 4-1). Atlassian has the resources to recruit users anywhere in the world, and they still go to where their users are to get their input.

In 2020, when we were writing this book, going to where your participants are gained a new meaning with the COVID-19 pandemic. Lockdowns, quarantines, and curfews suddenly made videoconferencing a daily necessity. In many cases, it was not possible to have a face-to-face conversation when businesses couldn’t safely open. A lot of the public spaces that speed up recruitment were not accessible; some may even have ceased to exist. While these changes pose challenges, it is still possible to be close to your participants. You can inhabit the same digital space through social media, collaboration platforms like Slack or Microsoft Teams, online discussion communities like Reddit, private chat groups on messaging apps like WhatsApp and Telegram, and massively multiplayer online games like Fortnite and PlayerUnknown’s Battlegrounds (PUBG).

Atlassian researchers working with users in a conference (source: https://oreil.ly/oBAo2)
Figure 4-1. Atlassian researchers working with users in a conference (source: https://oreil.ly/oBAo2)

Talking with your participants online will be different from face-to-face conversations, but with some up-front preparation, remote research can produce a similar quality of insights as face-to-face engagements. In fact, sometimes not sharing the same space with your participants makes them more comfortable sharing intimate, sensitive details with you. (We will cover remote research in Chapter 5.)

A considered selection of participants also gives you something else: a multitude of viewpoints on what you are exploring.

Genchi Genbutsu

Genchi Genbutsu is a Lean practice for diagnosing problems in manufacturing by personally going to the site of the problem and making firsthand observations.2 The Japanese term translates as “go and see for yourself.”

Seeking Different Perspectives

It’s important to seek different viewpoints when you are doing research. This can be done very easily by diversifying your recruitment to include current, potential, and extreme users.

Current users

Current users already use your product. They are important because they have experienced the product and can tell you how it affects their life. Their input will reveal common usage patterns, as well as what they like and don’t like about your product. More importantly, they will share issues they are having that you may not have realized.

Potential users

Potential users have expectations about your product, and they may have already satisfied those expectations through a competitor’s product. Or they may have tried your product in the past and left because you didn’t offer something that made them stay. This is an opportunity to learn from them.

Extreme users

Current and potential users are great sources of insight directly related to your product or service, but you can gain surprising insights from deliberately recruiting extreme users, or those with extreme behaviors. For example, if you are running a study on how amateur photographers compose their photos, an extreme user could be a pro photographer with a degree in cinema. Extreme users offer perspectives that challenge your design decisions and force you to reconsider some of your assumptions.

Of course, working with extreme users requires you to understand what constitutes “normal.” This understanding will come from your work in screening for “normal” users. Running a study with just extreme users can reveal a lot of new insights, especially if you are trying to answer a generative research question. However, this is risky.

We recommend combining “normal” users and extreme users to ensure that you have a comparative viewpoint. There is no global rule for this ratio, but we believe we achieved a good balance in our studies when we had one extreme user in a study of 7 to 10 people.

The Dynamic Duo: Researcher and Notetaker

If your research requires talking directly with participants, we highly recommend working in pairs: a researcher and a notetaker. The researcher’s role is to connect with the participant on a personal level and make sure the conversation keeps flowing. They establish a genuine connection, unique to that person, building a rapport with the participant. The role that is often overlooked is that of the notetaker. Yet this role is surprisingly key in the research process. In fact, the notetaker has three important responsibilities:

Capturing the conversation
The notetaker is responsible for taking notes and maintaining recording equipment. Taking notes does not mean writing everything down verbatim. Notetakers record salient, relevant points during the conversation, which then act as indices for capturing the conversation. The fact that the notetaker is taking notes does not mean that the researcher should not write anything down; instead, the researcher can simplify their notes around the flow of the conversation, such as themes to come back to or follow-up questions, and leave more specific notes to the notetaker. There may be situations where audio or video recording is possible. This is context dependent, as some participants may be less candid when being recorded.
Supporting the interviewer
The researcher is the first person to be exposed to the emotional responses of the participant, and for certain topics and participants, this can be taxing. The observer can maintain a distance from these responses and help the researcher keep their focus by taking over the questioning role at certain times, following the field guide just like the researcher does. They can also help with follow-up questions. Sometimes the observer is called the “second interviewer”: they can chime in, ask a question, and support as needed.
Seeing the wider context
The researcher’s main task is to maintain a personal connection with the participant while working on them to find answers to the research question. This is more difficult than it sounds. Because the researcher gives the participant their full attention, it’s inevitable that they will miss important cues in the environment. It’s up to the notetaker to capture these details in the notes and, if necessary, prompt the participant about them.

You may have noticed that the notetaker’s role requires them to talk to the researcher! Indeed, it’s a myth that observers should be quiet and not ask questions—far from it. The researcher-notetaker relationship is closer to that of a race-car driver and codriver: the driver holds the steering wheel, but both are involved in directing the vehicle. The driver may have more responsibility for moment-to-moment navigation, but the codriver has a greater awareness of their progress. The codriver’s hands-off role also gives them more cognitive bandwidth to analyze the course and suggest subsequent steps. If one person performs badly, the collaborative relationship allows them to recover from the situation, especially when they work as a team. Both drivers know how to operate the vehicle and can do it alone if they have to, but together they can achieve a lot more. The collaboration gets better the more they drive together. The same is true for researcher-notetaker pairs, who can perform well with little instruction. The important thing is the relationship: the more the pair works together, the better they get.

Another surprise: the roles of researcher and notetaker are interchangeable! In fact, it’s good practice for two people who are equally trained and invested in product research to alternate these positions. Note-taking can also be a good place to start if you’re new to research. You’ll learn about the flow of the research sessions, how to ask questions, and what to expect when something goes off script. Just remember that you should only change roles during the study itself if the researcher cannot proceed with the session in an appropriate way. Ideally, each study will have only one researcher and one notetaker. It’s possible to conduct a user study without a notetaker, but you may miss details during the session, which will force you to go back to review your recorded sessions later. The presence of a notetaker can make this step much more efficient.

We’ve talked about the researcher and the notetaker, but what about others? How many additional people should be listening to the participant in real time? We tend to keep that number to zero, maybe one. The participant is already outnumbered two to one, but at least both of the researchers are doing something. Adding a third or fourth person who just sits there can be distracting, and having more than one notetaker can be downright intimidating. All sessions should be recorded, if possible, for interested parties who want to review the entire session (remember to always obtain consent for audio and/or video capture). If the interested party is extremely concerned about being there in person, it is easy for them to help as the notetaker, not as an additional listener.

This rule applies to in-person and remote research studies. Recording is even more accurate for remote studies: those watching the recording are seeing exactly the same material the researchers saw. From the participants’ perspective, there is a difference between joining a video call with 2 people versus a call with 20 people watching and listening on mute. Remember that rapport and personal connection are key to getting good insights, and this connection is not possible in a crowded, impersonal call.

Preparing Guides for Your Sessions

If your research question is pointing you toward a qualitative method—one that requires you to talk to or interact with other people—a field guide is an essential tool to keep the conversation on track. A field guide condenses key information about the research and helps you to stay focused on the question at hand. It reminds you of your research question so that you don’t deviate from your objective. It also outlines 89the structure for the conversation, so you know where to go next. If different researchers are conducting multiple sessions for the same research project, a comprehensive field guide will ensure consistency.

Field guides are easy to prepare and helpful for all kinds of research sessions. For methods where you are having your participants do something, like a participatory design workshop or a usability study, the guide document is usually called a facilitation guide. We will talk about a few such methods in Chapter 6. For now, let’s look at a real-life field guide.

Example Field Guide

Magic: The Gathering (MTG) is a popular card-based fantasy role-playing game. The game is played in groups, and the large number of possible strategies makes each game unique. Doğa Aytuna, a PhD candidate at Kadir Has University in Istanbul, Turkey, studies MTG and the social constructs around it. Aytuna and his colleague Aylin Tokuç interviewed people who casually play MTG with physical cards (called “paper MTG”).

Let’s look at their field guide. You’ll note that the questions are not exhaustive. They are starting points for the conversation the researchers want to have. Their entire field guide fits on an A5 (or half-letter) sheet to make it easy to slip into a notebook and take notes on the page next to it. This is not a hard rule, but it helps the interviewers in note-taking and seeing where the conversation goes. Field guides are not sacred documents that should be hidden from the participants at all costs. They know that you have some questions for them. However, making the field guide distracting to the participants takes away from the conversation. The small, discreet size helps with that.

As we discussed in Chapter 2, be sure to dump your assumptions—or at least identify and examine them carefully—before you start. We’ll go through the process step-by-step, highlighting some important things to look for in this guide.

Drafting Your Field Guide

Write down your research question.

The research question is at the top as a reminder to the researcher. The research question itself is very focused and free of prejudices, as discussed in Chapter 3.

Brainstorm interview questions.

Use your preferred brainstorming or collaboration tool to come up with as many questions as you can. Here are a few tips for writing good questions:

  • Make sure your questions are open-ended and not leading.

  • Each question should have a single focus.

  • It’s a good idea to make sure each question is either attitudinal or behavioral, but not both. You will find it hard to get clarity on your results if you mix the direction of your inquiry.

  • Ensure that your questions are nonconfrontational. (In some cases, it may be appropriate to include a few well-considered provocative questions to promote thought and discussion.)

  • State your questions as impartially as possible.

  • Questions should be independent. Researchers should be able to ask them in any order based on the flow of the conversation.

Add notes.

Develop each question by annotating it in detail. You could add prompts and probes, such as follow-up questions seeking additional information. These will become your prompts (shown in italics in the example) to remind the researchers about things they want to follow up on, including topics toward which they want to nudge the participant.

These notes are best started with who, what, when, where, or why: “Why did that happen?” “When did this occur?” “Who else was involved?”

You can also make notes of things to be careful of, such as biases and preconceptions: “Don’t assume that the participant is a Catholic.” “Ask about the reason he is in the program.” “Note that the participant may be the primary caregiver of their parents.”

It’s also a good idea to put reminders about rapport: your connection to the participant. Are there things you can bring up to ensure that the participant feels listened to and understood? The way you talk to your participants can make or break the quality of the insights you receive.

Group your questions in themes.

Once you’ve developed the questions, group them together in themes. This will allow you to catch and eliminate any duplicates and ensure that your question set stays lean. The example guide has only three themes, with a few questions under each. More themes might make it hard to maintain focus, while more questions could feel like an interrogation.

Now it’s time to take your guide for a test run. As the 19th-century Prussian field marshal Helmuth von Moltke said, “No plan withstands the first contact with the enemy.” While your participants certainly aren’t your enemy, von Moltke had a point: no matter how thorough your field guide or how extensive your experience, the flow of your questions and your approach to inquiry will only improve once you’ve worked with your first participants. So treat the first few sessions of any research study as a pilot. Pick one or two users from the recruited participants and run the study end to end. This will help you finalize your questions and familiarize yourself with the kind of data you’re getting. Once you’ve completed the pilots, revise your questions. In extreme cases, you may realize that you need to change your recruitment strategy, so be prepared. In cases where recruitment is hard or you have such a small budget that you don’t want to risk using even a single qualified participant for a study that may fail, try to find a coworker or acquaintance whose attitudes and behaviors are as close to your target users as possible.

When C. Todd began to establish the design practice at MachineMetrics, one of the things he harped on with his team was “testing the test.” This meant that anytime they planned product research, they would do a dry run of any prototypes with internal users, often support or sales, before putting them in front of users. This was an opportunity for less-experienced team members to sharpen their user discovery skills. Like any skill, the more you practice, the better you get. Improving your skills can reduce the time you spend preparing and conducting product research.

Try It Out: Fixing a Field Guide

A hypothetical bank (we’ll call it the “Bank of Richness“) has hired a group of researchers to understand the experience of people applying for loans on their mobile app. One of the researchers wants you to review the questions they’ve come up with for their field guide before the interviews commence. Can you suggest any improvements? (We’ve provided some answers following the questions, but first come up with your own ideas.)

Mobile Loan Study

  • How much do you have in your accounts with us?

  • Do you currently have a loan?

  • What other products do you use from our bank?

  • How much do you pay for your mortgage monthly?

  • Are you an Android or an iOS user?

  • Do you use our app often?

  • Do you like our app?

  • How likely are you to recommend us to a colleague or friend?

Answers

The first pass is for looking at each question independently and fixing any issues with it. Is it a closed question? Does it lead the participant? Do you already know the answer? What follow-up questions could we ask to get richer responses from our participants? Here are some examples of notes we might make on the questions in this field guide:

How much do you have in your accounts with us?
The researchers can get this information from their data. By preparing thoroughly for the interview based on what you already know, as we described earlier in the chapter, you can avoid wasting precious interview time asking questions to which you already know the answers.
Do you currently have a loan? How much do you pay for your mortgage monthly?
These are closed questions, and they make it hard to get rich insights. Try turning them into open-ended questions, and make a note in the guide about how to expand on them with participants: for example, “What is your experience with loans?” Then probe for mortgage information, especially about the monthly payment.
What other products do you use from our bank?
Think about the language here. A banker or finance expert would know what a “product” is, but the average customer wouldn’t use this term. Most people don’t talk about banking “products.” They talk about savings accounts, credit cards, and loans. Choose familiar language rather than jargon to make your questions accessible. And don’t stop once your participants have shared which services they use. Make a note to probe them about their experience.
Are you an Android or an iOS user?

This is another question you could answer by examining your data before the interview starts. Alternatively, if you’re only interested in users of a particular operating system, use this question in a recruiting screener, not in the interview itself.

If mobile OS preference is relevant to your research question, try a more open-ended question, such as “What kind of mobile device are you using?” You could then ask how they chose their device to understand their preferences for mobile usage.

Do you use our app often?
This is a closed question and possibly something you could find in your usage data. If you want to hear about your participants’ mobile usage, a better way to ask this would be “How often do you use our app?” Then you could probe for recent tasks they’ve carried out and what their experience was.
Do you like our app?
Recall the confirmatory mindset in Chapter 1? This is a leading question, one that gently nudges the participant toward the answer you want to hear. A better way to ask would be “How would you rate our application?”
How likely are you to recommend us to a colleague or friend?
You may recognize this question as the Net Promoter Score question in Chapter 1. NPS has been widely used to measure customer satisfaction, but it’s a tricky question to get valuable insights from. A more straightforward question would be, “What are your opinions about our bank?” Then you could probe for moments of delight and ask if they have recommended you to anyone.

The second pass is for grouping the questions in themes. Here is a possible theme grouping for the questions in the field guide:

Experience with Bank of Richness

What are your opinions about our bank? Probe for moments of delight; ask if they have recommended us to anyone.

Which of our services are you using? Like savings, credit card, etc. Probe for their experience.

How often do you use our app? Probe for recent tasks they have carried out; ask how their experience was.

How would you rate our application, 1 being very bad, 5 being very good?

Loans on Mobile

What kind of mobile device do you use? Probe for the operating system or brands. Ask how they chose their device.

What is your experience with loans? Probe for mortgage, especially the monthly payment.

When you are done, look at the themes and the questions under each theme. Are the questions sufficient to tell you about what you want to learn for that theme? You can add and revise questions as you do this grouping for your project, because grouping under themes will help you think through your interview and make changes as required.

In this case, we think that the “Loans on Mobile” theme is a little weak. It asks the user about their smartphone choice and their experience with loans in general. We would recommend adding two questions that focus on user behavior:

  • Tell me about your last search for a loan. Probe for experience with the Bank of Richness and reasons behind their preference.

  • Can you walk me through how you applied for the loan? Ask if the user used a mobile app and if it was ours or a competitor’s.

Note how the last two questions ask the participant to recall a particular behavior rather than asking their opinions.

Creating a Communication Plan

Creating change requires collaboration. Keeping collaborators in the loop about what you are doing and when you would appreciate their help is a great step toward fostering collaboration. This is where a communication plan comes in handy.

A communication plan describes how new information should be communicated to the parties in a project. It includes types of activities that create new information, their frequencies, their outputs, parties who need to be informed about these outputs, their roles, their information needs, and the methods of communication to be used.

Communication plans can get incredibly detailed for large projects, but don’t let this intimidate you. You can put together a simple communication plan for your research that keeps everyone in the loop with three steps:

  1. Identify the communication groups.

    Most research projects have three groups that need to be kept in the loop. The first group includes those with active roles during the execution of the research. Anyone who will have direct contact with the participants, including the notetakers, belongs to this group. The second group includes the people who may be affected by the research and therefore should contribute to the analysis in some shape or form. Usually (and preferably), the first group is a subset of the second. The sponsors and senior influencers make up the third group. This group may include directors, key executives, and decision makers.

  2. Decide on the frequency and format of communication.

    Decide how frequently you need to communicate with each group. Note each group’s different needs and how it would benefit from different depths of information; people with active roles may appreciate a Slack channel with almost real-time updates about each participant session, whereas executive sponsors may look for a single, pithy, easy-to-read email about the process and final outcomes. Determine how you share updates with each group.

  3. Follow the plan.

    As each research activity takes place, follow the steps outlined in your communication plan. For large projects, loop in the project manager. Remember that communication is not a one-directional activity: be open to hearing back from the parties you are working with. They may ask questions, request additional information, or offer suggestions that could affect subsequent steps. If you are getting recurring requests of the same type, you might have missed a step in your plan. Take your time to revise your plan and continue your updates based on the improved version.

Having a plan does not guarantee communication. Remind yourself to follow your plan as you tackle each step of your research. If the parties are not responding when you feel they should, contact them through other methods to make sure that they are informed and to listen to their concerns. Take care not to be pushy, and try to understand their needs. Then, update your communication plan accordingly to keep the information flowing.

What If You Can’t Stick to Your Plan?

It’s not the end of the world.

Researchers prepare ahead of time to avoid methodological mistakes. We review our research questions to understand the context and business dynamics around them. We make our assumptions explicit to radically decrease our chances of bringing our biases into the sessions so that we can listen to our participants without prejudice, with the sole goal of understanding them and learning from them. We pick our participants deliberately to ensure that they provide useful feedback to us. We plan how we will capture the sessions and are ready to handle surprises as a team. We prepare guides in advance to keep us on track during our conversations with the participants. By thinking about these topics in advance, we decrease the possibility of methodological errors in our research.

Not having to worry about the methodological correctness of your activities gives you something extremely valuable: attention. Attention is critical in research where we come together with other people. Knowing that you have the right method, that you are working with the right participants, and that you are working together with a partner to capture the conversation relieves you from having to think about your research approach. This frees you to give your full attention to your participants. As you will see in Chapter 5, giving your undivided, unconditional attention to your participants is essential for creating the empathic connection necessary for understanding them.

Preparing for research is not about perfection. Aim for coherence and awareness about your research approach. It is not about pure consistency, either; it’s OK to modify your questions and prompts on the fly if you have to. We cannot count the number of times we’ve headed out with our research guide in hand, arrived at the client site, asked the first question, and then improvised as the conversation took an unexpected left turn.

Research planning is like preparing for a multicourse dinner party. You could start cooking when your guests arrive, but if you do, you’ll be too occupied with cooking to entertain your guests and distracted enough that the food might not be your best. It will be a more memorable, enjoyable event for everyone if you prepare ahead of time. That way, you can spend quality time with your guests and enjoy a delicious menu together.

All research sessions where we come together with participants, such as interviews, usability studies, or eye-tracking studies, follow a similar sequence of events. Getting familiar with this flow will help you arrive at insights effectively. This flow consists of knowing your research question and assumptions, finding users, working with them, and starting preliminary analysis while you are still working with your participants. (More on this in Chapter 5.) Having a communication plan around these activities helps you share your progress with colleagues contributing to research and with stakeholders who will be affected by your insights.

You will see that you get better at preparing for research the more you do it. We will talk in Chapter 9 about how you can turn research into a habit.

Rules in the Real World: How Can You Tell If People Feel Connected in a Video Call?

COVID-19 has changed the way we work. In 2020, as we write this book, we’ve had to confine our activities to our homes and find creative ways to do things together remotely over video calls: not just work and meetings but also birthdays, parties, and even weddings. While we can share the same amount of information, we lose the human texture of being together in the same space.

The team working on Microsoft Teams, a collaboration suite that offers chat, video calls, and file storage, has been exploring ways to make participants in video calls feel more connected and together. In developing a recent feature, they wanted to create the feeling of sitting together in a meeting room or having coffee around the break-room table.

Their solution was together mode, a special mode for multiperson video calls where the participants are placed virtually in simulated real-world settings (see Figure 4-2). For example, if you are teaching a class using together mode, the shared video feed looks like a classroom instead of a grid of separate videos.

Cool! But the team wanted to know whether the feature actually met their initial goal. Does together mode make people feel like they are actually sitting and working together? How do you measure togetherness?

To answer this question, they conducted a usability study where they monitored participants’ brainwaves while they worked. They recruited three groups for the study, around 20 people total. One group worked together in the same physical space, one group worked together in a virtual environment using the standard grid view, and one group worked together in a virtual space using together mode. Researchers found that the brainwaves of the team who used together mode were much closer to those of the team who worked together in a physical space, compared with the ones who used the grid of videos to collaborate.3

Microsoft Teams’s new offering, together mode
Figure 4-2. Microsoft Teams’s new offering, together mode

The method choice is important here. Microsoft could have saved a lot of money by doing a remote usability study with hundreds of users instead of monitoring the brainwaves of a small group. Or they could have sent out a survey to thousands of people to learn about how they feel when they use together mode. A remote, unmoderated usability study or survey would be much easier to run than a usability test with complicated instrumentation; it would also be cheaper, and they could reach a much larger crowd.

Why did Microsoft researchers not use these methods? Because these methods would not have given them a reliable answer to their question. Self-reported methods, such as surveys or remote, unmoderated usability studies, could have yielded some results at a fraction of the cost, but they would not be reliable enough for the specificity they needed. The team was interested in very specific feelings under specific circumstances. So they chose a method that allowed them to study cognitive signals. They could answer their research question with high confidence.

Key Takeaways

  • Preparing for research is essential for success. While some improvisation is OK, you can’t skip preparation completely, no matter how experienced a researcher you are.

  • You’ll pick a research method based on where you are in the product development process and what you want to learn from your participants. Also consider your own research skills, the cost of the method, and the cost of recruitment.

  • Be careful not to select the easiest method at your disposal; it may not provide you with the answers you seek.

  • A diverse participant group that is motivated to provide feedback is invaluable. You can put together such groups easily by going to where they are, physically or digitally.

  • Use screeners to determine whether people are suitable for your study.

  • Working in researcher-notetaker pairs makes research a lot more effective and fun.

  • Communicate your progress to everyone who is interested in or affected by your research. This makes it easier for others to contribute to your research and for you to turn insights into actionable items.

  • If things don’t go according to your plan, it’s not the end of the world. There will be times when you have to start working with users very quickly. Spend extra time reflecting on your process in these cases, so you can learn from your mistakes and adjust during analysis if necessary.

How well does your team make personal connections and develop empathy for and with users?

1 We found Sam Ladner’s book Mixed Methods: A Short Guide to Applied Mixed Methods Research (https://www.mixedmethodsguide.com) and Christian Rohrer’s taxonomy of research methods (https://oreil.ly/tyhK8) particularly useful for our own work.

2 “Toyota Production System Guide,” The Official Blog of Toyota UK (May 31, 2013), https://blog.toyota.co.uk/genchi-genbutsu.

3 You can read more about Microsoft’s related research on remote collaboration (https://oreil.ly/qEWGa). You can read about together mode’s development (https://­oreil.ly/ttcMq).

Get Product Research Rules now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial