Cognitive testing CIS 2013:Methodological guidelines

Petra Andries, Anneleen Bruylant and Machteld Hoskens

Presented to the representatives of the Belgian Federal Science Policy Office and of the Direction Générale Opérationnelle de l‘Economie, de L‘Emploi et de la Recherche de la

Région Wallonne on March 7, 2012

What to expect

• Some background• General Guidelines• Examples• Putting theory into practice• Our CIS question• Observations and Discussion

Some background

• Willis, Gordon B. (2005). Cognitive Interviewing: A tool for Improving Questionnaire Design

• Van Cruysen, Adriana and Arundel, Anthony (2010). Guide to conducting cognitive interviews of survey questionnaires

General goal

• Obtaining insight in possible problems (both overt and covert) a questionnaire might have.

Major sources of error

• Errors of non-observation: Are we surveying the right people?

• Errors of observation: Are we getting the right answer?

• Post-observation error: Are we doing the right things with the data?

Are we getting the right answers?

• Interview error: Interviewer reads the question incorrectly, or registers the answers incorrectly, …

• Response error: Question characteristics and the way respondents process the question

Question characteristics

• Conflicting, inaccurate or complicated instructions• Question is lengthy, awkward, ungrammatical, or contains

complicated syntax• Technical terms are undefined, unclear or complex• Vague question: there are various ways to interpret the

question or to decide what is to be included or excluded• Reference periods are missing, not well specified or in conflict

Assumptions

• Inappropriate assumptions are made about the respondent or about his/her situation

• Assuming constant behavior/experience for situations that vary

• Double-barreled questions: contains more than one question

Respondents’ processing• Respondent’s may not know the answer to the question, or

may not remember the information asked for• They may not have the attitude/be in the situation they are

being asked about• The question may require a difficult mental calculation• The question may ask about sensitive issues• Socially acceptable responses may be implied• Overlapping response categories

General Guidelines

• Generally, a relevant sample is chosen between 5-15 respondents

• Trained interviewers are used to conduct the test• If problems become apparent early on during the testing,

changes can be made to the questions accordingly, and testing continues using the altered version

• Think-aloud technique: the subject has to report everything he/she thinks about as they work their way to an answer

• Verbal probing: the use of probes to determine whether the subject understands the question as intended, if the different technical terms are understood, etc.

• A combination of both techniques will be used during our testing of the questions

General Guidelines (ii)

• The interview may never exceed an hour• Keep written notes of the interview, even if the interview is

being recorded, and record what time you start and end the interview

• Prepare the probes to be used, but feel free to ask any additional questions you may have, but don’t feel like you have to probe everything

• This exercise is meant to find out whether the questions obtain the information we aim for, whether the questions are understood as we designed them, whether any part of the question suggests bias, etc.

General Guidelines (iii)

• Interviewers have to understand the questions themselves, so they are able to answer any questions the respondents may have

• Keep your introduction short• Keep the pace of the interview relaxed, but steady• Do not defend or criticize the questions, and keep the tone of

your voice neutral, as well as the information you give (so as not to bias the respondent)

• Probe proactively, even if there is no overt sign of trouble

General Guidelines (iv)

• Interviewers have to stress the fact that they did not design the questions themselves, so any criticism is welcome

• Our goal is to find overt and covert problems of the questions, so we are not interested in the actual answer the respondent gives

• Start the interview with a short introduction, mentioning why we conduct this test, and explaining what will happen during the test. Use the windows-question to see whether your instructions are clear to the respondent. End the interview with thanking the respondent for his/her time, and offering the opportunity to mention anything else he/she feels like mentioning

General Guidelines (vi)

Examples

• Q: How many times have you talked to a doctor in the last 12 months?• A: I guess that depends on what you mean with “talked”. I talk to my

neighbor, who is a doctor, but you probably don’t mean that. I go to my doctor about once a year for a general checkup, so I would count that. I’ve also probably been to some type of specialist a couple more times in the past year – once to get a bad knee diagnosed, and I also saw an ENT about a chronic coughing thing, which I’m pretty sure was in the past year, although I wouldn’t swear to it. I’ve also talked to doctors several times when I brought my kids in to the pediatrician – I assume you don’t want that included, although I can’t really be sure. Also, I saw a chiropractor, but I don’t know if that’s a doctor in the sense you mean. So, what I’m saying, overall, is that I’m not sure what number to give you- not sure what you want.

• Q: How often do you, or the person who shops for your food, buy items that are labeled ‘low salt’?

• Problems: – One subject thought of low fat, not low salt. – Several subjects said ‘yes’, but they meant that they get foods that they know are low in

salt, but it isn’t necessarily mentioned on the label.– Different people have different behaviors (‘My wife always checks, I never do.’)– How do you interpret an answer as ‘seldom’? (they rarely buy the types of food that

contain sodium, or they buy lots of such foods, but not the low-sodium ones.)– One subject just read the mg of sodium, does that count as a label of ‘low salt’?

• Probes:Who shops for your food?Tell me in your own words how (you/they) decide how to choose foods that contain salt.

Examples (ii)

• Suggested rephrasing of the question:

Q: THE NEXT QUESTIONS ARE ABOUT SALT OR SODIUM IN FOODS.a) Who shops for most of the food you eat: you or someone else?b) When you/they shops for food, how often do you/they buy low-salt

or low-sodium varieties of foods, when you/they have a choice?c) How do you/they usually select a food that is low in salt? Does it say

‘low salt’ on the label, or do you/they just know it is low in salt?

Examples (iii)

Putting theory into practice: intro

• Thank you for taking out the time to see me. First of all, I would like to explain why we are doing this exercise. We are testing some questions that have given us quite a few problems, as quite a few companies seem to answer them incorrectly. We suspect the questions are not understood by the companies the way we intended them.

• To give you an idea of what we’re going to do: I will ask you two questions, which you will answer by telling me out loud everything that crosses your mind. I may interrupt you sometimes with additional questions, and I will be writing down your answers. Do not hesitate to ask for clarifications if something is not clear, or if it is too difficult to answer. I did not design the questions myself, so any comment or criticism is welcome! Also, we are not interested in the answer you give in itself, but rather, we want to discover whatever is wrong with these questions, and why companies have a hard time answering them.

• We will try out a simple question first, so you get an idea of what the test will be like:– Imagine you are at home. Now tell me how many windows there are. As you count the

number of windows, tell me what you see and what you’re thinking about.– Was this question: very difficult, difficult, a little difficult, or not difficult at all? Why?

Putting theory into practice: Getting warmed up

Putting theory into practice: Our CIS questionCombien votre entreprise a-t-elle dépensé en 2012 pour les activités ou les achats suivants? Veuillez également donner une estimation de la part de ces acquisitions (ou d’activités) qui était prévue pour l’innovation de produits ou de procès comme défini ci-dessus. Activité Dépenses 2012 Estimation de la part de l’activité

prévue pour l’innovation de produit ou de procès

• R&D interne _____________€ ____% (y compris les dépenses en capital relatives aux bâtiments et aux équipements spécifiquement consacrés aux activités de R&D coûts de personnel et coûts liés)• Acquisition de R&D externe _____________€ ____%• Acquisition de terrains, édifices, _____________€ ____% machines et équipement (à l'exclusion des dépenses pour des équipements destinés à la R&D)• Acquisition de logiciel _____________€ ____%• Acquisition de connaissances externes _____________€ ____% (y compris licences)• Formation payée par l’entreprise _____________€ ____%• Publicité, campagnes de publicité, _____________€ ____%• études de marché• Activités de design (contrats interne et _____________€ ____% externe)

• What problems can you foresee? What might not be understood?

• Suggestions for probes we can ask? Remember, probes are more effective if they are specific, rather than diffuse and general!

• Let’s try it out!

Observations and discussion

• Observe as we try out the questions on each other.• Any questions?