By Darby Steiger, MA, MPP
Vice President
Innovation & Solutions, SSRS
Glen Mills, Pennsylvania
dsteiger@ssrs.com
By Andrew Stavisky, PhD
Assistant Director
Applied Research & Methods, U.S. Government Accountability Office
Washington, DC
staviskys@gao.gov
Imagine you are a qualitative researcher working with a consumer brand that is preparing to launch a large customer experience survey. The research team has already drafted the questionnaire internally and asks you to run a few exploratory interviews with customers to “get a feel for how people think about the brand.” During one of your interviews, you test out a question from the draft survey: “How satisfied are you with the value you receive from the product?” Your first participant pauses and asks, “What do you mean by value—price, quality, or how often I use it?” Another participant answers quickly but later explains that she interpreted “value” as whether the product feels worth the effort it takes to use it. A third participant says he chose a middle rating because the price feels fair, but the product doesn’t last as long as he expected. What looked like a single, straightforward concept to the research team—value—actually means different things to different customers.
Moments like this may happen frequently across market research: a loyal customer may be turned off by the messaging in an upcoming ad campaign; a potential customer gets lost in the navigation of a new feature on a website; or a new product user is asked for feedback on their experience and does not understand the wording of a question on the customer feedback survey. Learning the practice of cognitive interviewing can aid in turning those moments into a more deliberate practice to help uncover how consumers or users are interpreting materials, websites, or survey language, what experiences they are drawing on when interpreting the intended information, and whether the language of the materials truly matches how respondents think about the topic.
Even when a project doesn’t include a formal cognitive interviewing phase, these skills can add immediate value for qualitative researchers. A moderator who understands the cognitive psychology behind how people consume and interpret information can listen for signs that materials may be misunderstood or interpreted inconsistently. Sharing these observations with clients can help refine survey or material wording before launch—often preventing costly problems later. For qualitative researchers who primarily conduct focus groups and IDIs, cognitive interviewing is therefore not a separate specialty so much as an extension of the skills already applied. Cognitive interviewing challenges us to listen carefully, probe strategically, and be diligently curious about how people interpret information.
What is Cognitive Interviewing?
Cognitive interviewing is a qualitative research technique designed to understand how people interpret and respond to questions, instructions, or materials. At its core, cognitive interviewing asks two deceptively simple questions: “What’s going on in people’s minds when they answer a question or interact with a product or material?” and “If it is not in alignment with what the researchers expected, what is the source of the disconnect?”
The technique is built on a framework developed by cognitive psychologists in the 1980s called “Cognitive Aspects of Survey Methodology” or CASM. The basic idea is that survey respondents go through four cognitive steps to answer questions, and that understanding and testing these mental processes can greatly improve the accuracy and validity of survey data (see Figure 1).
- Comprehension: Do they understand the question as intended?
- Retrieval: Can they access the necessary information from memory?
- Judgment: Can they evaluate and formulate an answer?
- Response: Do the available survey options provide a match for how they want to answer?
Problems can occur at any of these stages, and cognitive interviewing systematically tests for issues in each one.
Figure 1. The Four-Stage Cognitive Framework.
Why Cognitive Interviewing Matters Now: The AI-Resistant Skill
Artificial intelligence is rapidly transforming qualitative research. Large language models can transcribe interviews, identify themes, and generate insights at scale. What AI can’t do, at least not yet, is understand the nuanced, in-the-moment cognitive processes that cognitive interviewing reveals.
When you sit across from someone and ask, “What do you think this question is asking?” or “How did you come up with that answer?”, you’re accessing something fundamentally human: metacognition (the ability to reflect on one’s own thinking). This makes cognitive interviewing uniquely valuable in an AI-saturated research landscape.
The Technique in Action: Three Approaches to Probing
The heart of the cognitive interview is the probe—the follow-up questions we ask to understand how people are processing a question or material. Some probes are scripted, targeting words, concepts, or questions we’ve already hypothesized might cause trouble. Others are spontaneous, triggered by verbal hesitation or a puzzled expression we weren’t expecting. Whether scripted or spontaneous, probes fall into three general styles (see Figure 2).
1. Concurrent Probing: Real-Time Feedback
In concurrent probing, the researcher pauses after specific questions to ask follow-up probes in the moment, while the participant’s thoughts are still fresh. It is best used in the development phase for surveys, products, materials, or websites.
Example probes:
- “What do you think is meant by [word or phrase] in this document/material?”
- “What were your initial reactions to this language/design?”
- “How easy or hard was it to decide on your answer/figure out what to do next?”
- “I noticed that you hesitated. Tell me what you were thinking.”
2. Retrospective Probing: Preserves the Natural Flow
In retrospective probing, the interviewer waits until the end of a survey section or web page before asking participants to think back and reflect. It is best used when the questionnaire or material is closer to final form so we can capture the performance of the entire document or website.
Example probes:
- “Looking back, were any questions hard to answer? Looking back on the whole document/website, what stood out for you?”
- “How did you feel about being asked questions about [TOPIC]? How did this document/information make you feel?”
3. Think-Aloud: Unfiltered Thought Process
In think-aloud probing, participants are asked to verbalize their thinking as they work through each question or material—the interviewer simply listens. It is especially effective for complex questions or tasks and can be combined with concurrent probing.
Example instruction:
- “As you read and answer the question, please tell me everything you are thinking about as you decide on your answer. As you review these materials, please tell me everything you’re thinking about.”
Figure 2. Comparing Cognitive Probing Techniques.
Cognitive Probing Techniques: Putting It All Together
In practice, many cognitive interviews blend more than one of these approaches. Concurrent and think-aloud probes are used to capture in-the-moment reactions, with the addition of retrospective probes to surface higher-level impressions about an entire survey or document.
In a typical cognitive interview, we focus our concurrent probing on questions or language that we suspect may contain difficult concepts or terms, ideas that are open to interpretation, questions that are asking about events that may be challenging to recall, questions that may feel burdensome, or questions that include a list of response options that may be incomplete. As participants move through the survey or materials, we pause to probe on that specific language, using spontaneous probing for any other items where the participant hesitates or seems confused. At the end, retrospective probes catch any issues we may have missed.
The Power of Cognitive Interviews
The following two case studies illustrate how just a few simple cognitive probes can reveal issues with even the simplest survey questions and provide guidance for how to improve them.
Case Study #1: Car Ownership
- Original question: “Do you own a car?”
- Sample Cognitive Interview:
- Interviewer: How did you decide on your answer?
- Participant: Well, I wasn’t quite sure how to answer. My wife drives a pickup truck. Do trucks count as cars? And technically, we don’t own it, she leases it. And it’s my wife, not me, so I wasn’t sure whether you wanted me to include her.
- Cognitive issues revealed: Comprehension (household scope, ownership types, which vehicles count)
- Improved version: “Do you or any other adult in your household own, rent, or lease any motor vehicle, including a car, truck, van, or motorcycle?”
Case Study #2: Alcohol Consumption
- Original question: “How many alcoholic drinks have you had in the past 30 days?”
- Sample Cognitive Interview:
- Interviewer: How did you decide on your answer?
- Participant: That’s a tough one! I am not sure I can remember how much I drank over the past 30 days, and I’m not sure I’d really want to tell you that anyhow because it was a lot!! Could I just tell you about the past week, which wasn’t as bad?
- Interviewer: How would you answer it for the past week then?
- Participant: Well, let’s see. I had two glasses of wine last night. But is wine really alcohol? Then I had a sip of my friend’s drink last weekend—does a sip count? And then I had about half of a beer while watching football. Hmmm, does that count as a whole drink? Now I’m confused.
- Cognitive issues revealed:
- Comprehension: What counts as “alcohol”? Does beer count the same as liquor?
- Comprehension: What constitutes a “drink”? A bottle? A glass? A shot?
- Retrieval: Can I accurately remember how many drinks I had over 30 days, and how precise do I need to be?
- Judgment: What will researchers think if I tell the truth? Social desirability pressure pushes toward underreporting.
- Improved version: “In the past seven days, how many drinks of an alcoholic beverage have you had? Count as a drink: a can or bottle of beer; a wine cooler or a glass of wine, champagne, or sherry; a shot of liquor; or a mixed drink or cocktail.”
Practical Tips for Conducting Cognitive Interviewing
Sample Size and Recruitment
We typically conduct testing with eight to fifteen participants to gather the feedback needed to determine whether the survey, materials, website, etc., are ready for launch. After this point, we often find ourselves reaching a saturation point where we are hearing the same types of problems. Participants are screened to ensure they reflect the target population of the survey.
For example, if we are testing a survey about experiences with health plans, we conduct cognitive interviews with people who are insured. We also try to include a diverse mix of participants within that population to ensure the materials will perform well for all types of respondents. For example, we might want to include a few who are less familiar with the concepts in the materials, or who have lower levels of education, to make sure the concepts will make sense.
Finally, if time and budget allow, we often will conduct multiple rounds of cognitive interviewing. We conduct a few interviews, make some adjustments, conduct a few more, etc., until we are confident that the questions or materials are performing well.
The Cognitive Protocol and Interview
The cognitive interview protocol generally includes the following elements:
- Welcome & logistical information
- Description of the interview goals: explaining that the participant will be asked to complete a questionnaire or review materials while the interviewer quietly observes and occasionally interrupts to ask follow-up questions
- Accessing the materials: instructions for providing the participant with access to the survey or materials to be tested
- Suggestions for spontaneous probes: guidance on what to ask if the participant has feedback on items or language not included in scripted probes
- Scripted probes: a list of scripted probes that are to be asked after selected questions or sections of the materials, and/or after reviewing the entire document, website, or set of materials
- Thank you and provision of incentive
A typical cognitive interview can run anywhere from 30 to 60 minutes, depending on the length of the survey, amount of materials, or web screens being reviewed. The flow of the interview usually looks something like this:
- Introduction (5 minutes): Build rapport, explain the process.
- Observation and probing (20-45 minutes): Quietly observe as the participant reviews materials, pausing to probe with both scripted and spontaneous probes.
- Debrief and closing (5 minutes): Overall impressions, anything missed, thank you, and provision of incentive.
Analysis Approach
Cognitive interviewing analysis is highly structured: for each question, material, or web page, researchers document any issues or problems that arose based on the probes. This might include comprehension failures (misunderstood/misinterpreted terms or complex instructions), retrieval challenges (difficulty remembering, or a timeframe that doesn’t match the construct being asked about), judgment difficulties (having a difficult time coming up with an answer, or reluctance to answer a sensitive question), or response mismatches (missing or overlapping categories). In the analysis, researchers qualitatively note how frequently the issue arose (most, some, a few, once). Each finding should map directly to a recommendation or explicitly note if no change is recommended.
As a reminder, cognitive interviews can’t quantify prevalence. If 3 of 12 participants misunderstand a term, you know there’s likely a problem—but not whether it will affect 3 percent or 30 percent of your full population. Use cognitive interviewing to identify problems, then validate with pilot testing.
Best Practices: Probing Dos and Don’ts
Cognitive interviewing skills align closely with our skills as qualitative researchers.
- A basic understanding of the four-stage cognitive processing framework and questionnaire design principles (if evaluating a survey)
- Grounding in the goals and intentions of the survey/materials being tested
- Ability to quickly establish rapport and trust to encourage honest feedback
- Active listening and awareness of nonverbal cues
- Ability to ask strategic and neutral probes
- Awareness of timing to maximize how many questions or materials can be reviewed during the interview
Below are some additional tips to be a successful cognitive interviewer:
Do:
- Remain neutral in verbal tone and body language.
- Use balanced probes: “How easy or hard was that to understand?”
- Probe to understand the problem, not solve it.
- Record sessions for detailed analysis later.
- Test in the same mode as final deployment when possible.
Don’t:
- Correct participant errors or try to help them understand what the material is presenting.
- Use biased probes: “Was that hard to understand?” (assumes difficulty).
- Ask participants to paraphrase (“In your own words, what is this saying?”)—this tests verbal skills, not comprehension.
- Ask participants to fix the question or document—that’s your job.
- Claim statistical significance from small samples.
What Cognitive Interviewing Unlocks
Beyond the traditional use of cognitive interviewing to improve survey questions, cognitive interviewing opens new opportunities for qualitative researchers. It can also be used for UX and usability testing—website navigation, form completion, product interfaces—and connects seamlessly with pilot testing and quantitative validation for mixed-methods teams. Perhaps most importantly for business development, it creates a service that clients don’t know to ask for. Most clients never think to test their customer satisfaction surveys, advertising or marketing materials, or websites, and you can proactively offer this preventive work before costly mistakes happen.
Final Thoughts: An Investment That Pays Off
When we tell clients about cognitive interviewing, a common response is: “Why haven’t we been doing this all along?” The technique seems obvious in retrospect—of course you should test materials before implementation. Yet remarkably few organizations build cognitive interviewing into their standard processes. This creates an opportunity for forward-thinking qualitative researchers to add genuine value, prevent costly mistakes, and position themselves as indispensable partners.
In a world where AI can do increasingly more of what we do, cognitive interviewing represents something fundamentally human: the ability to watch someone think, to notice when they’re confused, and to probe until you understand the nature of the problem. It’s messy, nuanced, and deeply relational—exactly the kind of work that makes human researchers irreplaceable.
So, next time you’re handed a questionnaire, a form, or any material meant to elicit reactions, ask yourself: “What might happen when real people encounter this?” Then grab your cognitive interviewing toolkit and find out.
References
- Willis, Gordon B. Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage Publications, 2004.
- Beatty, Paul C., and Gordon B. Willis. “Research Synthesis: The Practice of Cognitive Interviewing.” Public Opinion Quarterly 71, no. 2 (2007): 287–311.
- Miller, Kristen, Valerie Chepp, Steven Willson, and José Luis Padilla, eds. Cognitive Interviewing Methodology. Hoboken, NJ: John Wiley & Sons, 2014.
- Tourangeau, Roger, Aaron Maitland, Daniel Steiger, and Ting Yan. “A Framework for Making Decisions about Question Evaluation Methods.” In Advances in Questionnaire Design, Development, Evaluation and Testing, edited by Paul L. Beatty, Deborah Collins, Lyn Kaye, José Luis Padilla, Gordon B. Willis, and Amanda Wilmot, 47–73. Hoboken, NJ: Wiley, 2020.
About the Authors
