Kristen Miller, Ph.D. Question Design Research

Office of Research and Methodology

National Center for Health Statistics

Question Evaluation at the National Center for Health Statistics

Data come from Questions

Bad questions can mess up your findings

(in potentially embarrassing ways)

Women: BMI by ‘Sexual Orientation’

2002 NSFG

HETEROSEXUAL HOMOSEXUAL BISEXUALNormal 49.7 30.5 48.5

Overweight 25.6 35.3 29.6

Obese 24.7 34.2 22.0

2006 NSFG

STRAIGHT OR HETEROSEXUAL

GAY OR LESBIAN OR HOMOSEXUAL BISEXUAL

Normal 40.7 38.2 36.5

Overweight 26.5 33.0 19.5Obese 32.8 28.8 44.0

Data come from Questions

It is critical to understand how questions perform to understand what the data really are.

Why question evaluation is important…

Goals for Question Evaluation:

How do the respondents understand the survey question?

Do respondents understand the survey question differently?

Does the question mean the same in all the languages that it is asked?

Does the question mean the same in all culture and socio-economic groups that it is asked?

In processing a question, do all respondents recall information and form an answer the same way?

Goals for Question Evaluation:

To what extent are the data elicited from the question a true representation of the phenomena being studied?

In what ways is our picture distorted because the questions do not accurately capture the intended construct?

What important discovery are we not making because we are unaware that our picture is distorted?

This Presentation

Describe QDRL and Question Evaluation at NCHS

Discuss question evaluation and cognitive interviewing

Examples along the way

Question Design Research Lab

National Center for Health StatisticsOffice of Research and Methodology

FacilityLab (room with video cameras, microphones)Video data baseSoftware application for analyzing cognitive interviews

ActivitiesApplied Testing Projects (Cognitive Interviewing Studies)Research related to question evaluation methodology

QDRL Guiding Principles1. The best question design is based on

question evaluation, not expert opinion

2. Question evaluation is science-based Empirical Evidence Transparent and Systematic analysis

3. Evaluation studies must be well documented, replicable, and made accessible

Developments include:

Focus on analysisCombining various methods to improve

explanatory power of question performanceIntegration of qualitative and quantitative

methodologiesFocus on measuring comparability across

subgroups/disparity groups With an eye toward quantifying measurement

errorDevelopment of application suite:

Q-Bank, Q-Video, Q-Notes, Q-Notes Plus

A good question is…1. relevant to the research agenda

and

2. relevant to each potential respondent’s experience and knowledge.

What to Remember about Respondents

Do not know or understand the research question

Most likely, do not use scientific, abstract concepts

Survey puts them in the position of operating as informants

Reference aspects of their lives

When this relationship is broken, error is introduced into the data.

False PositivesFalse NegativesAn entirely different phenomena is measured

than intended by the research agendaExample: Terrorism

Question Response Process

RetrievalComprehension Judgment Response

Social FactorsSocial Factors

Social Factors

Social Factors

Social Factors Social Factors

Social Factors

17

Expectations of a survey respondent:

An impromptu response is required

Responses must be categorizeable

18

Expectations of a survey respondent:

An impromptu response is required

Responses must be categorizeable

Questions will be understood using formal language structure

19

Expectations of a survey respondent:

An impromptu response is required

Responses must be categorizeable

Questions will be understood using formal language structure

The analytic purpose of questions will be intuited and responded to accordingly

20

Expectations of a survey respondent:

An impromptu response is requiredResponses must be categorizeable Questions will be understood using

formal language structureThe analytic purpose of questions

will be intuited and responded to accordingly

Meaning to vague words, such as “mild,” “moderate,” “severe,” “extreme,” will be assumed through ordering

21

Expectations of a survey respondent:

Inside the cultural frame of reference

Inside the questions’ system of knowledge

22

Chronic Conditions

23

Questions requiring an answer outside respondents’ knowledge

Chronic condition questionsMedicine questionsMedical test questions

Cognitive InterviewsDesigned to understand how respondents

comprehend, retrieve, judge, respond to questions

Through this examination, can identifypotential response errors patterns of interpretation

Provide insight into social-cultural factors that impact the response process

Cognitive Interviewing Study

QualitativeSmall sampleSample selection purposive Examines thought processes of

respondentHow does the question work?Does the question work as intended?

If not, how can it be “fixed”?

Cognitive Interviews are Semi-structuredCore Question- interview is organized by

the questions that are being tested

Probe Questions- open-ended, spontaneous, not pre-scripted, based on the information that the respondent provides

Probing for storyWhy did the respondent answer the

question the way that they did?

Does this story match with the intent of the question?

Data from Cognitive TestingCollected from semi-structured protocolNarrative formatValidity tied to rich detailFindings are groundedInsight into question interpretationInsight into patterns of calculation

Cognitive Interview FindingsProvide knowledge of question

performanceIllustrate what the question measures

Varied patterns of interpretationDimensions of response error

Overall, during the past 4 weeks, how much difficulty did you have with thinking clearly and solving daily problems?

Respondent 5

Respondent 2

Respondent 6 Respondent 3

Respondent 4

Respondent 1

Alzheimer’s disease

Busy

Long term, medical problem

Specific experience- organizing tennants

Remembering detailed list

Fiscal functioning

Cognitive Interviews

Designed to understand how respondents comprehend, retrieve, judge, respond to questions

Through this examination, can identifypotential response errors patterns of interpretation

Provide insight into social-cultural factors that impact the response process

Cognitive interview and question evaluation reports written and housed on Q-Bank

Some Purposes of Cognitive Some Purposes of Cognitive InterviewingInterviewing1. Ensure the question is capturing intended

construct (validity) and document construct for data user

2. Hone construct so that it is meaningful and measureable

3. Assess comparability of translations4. Assess comparability across socio-cultural and

demographic subgroups (e.g. countries, SES status, age)

5. Determine/Confirm meaningful thresholds or cut-off points

6. Assess how similar questions work together (e.g. frequency and intensity questions)

“Case Study”

UNESCAP/Washington Group collaborationCambodia, Kazakhstan, Maldives, Mongolia, Cambodia, Kazakhstan, Maldives, Mongolia,

Philippines, Sri LankaPhilippines, Sri Lanka

To examine an extended set of disability questions with intent to develop measuresMobility, Vision, Hearing, Pain, Affect,

Fatigue, Cognition, Learning, Communication

Intended for global use

This Study: Mixed Method This Study: Mixed Method ApproachApproach

A) Cognitive Interviewing: A) Cognitive Interviewing: Qualitative and Semi-structuredQualitative and Semi-structured143 interviews143 interviews

B) Field Interviewing: Quantitative B) Field Interviewing: Quantitative and Structuredand StructuredApprox. 1000 interviews per Approx. 1000 interviews per countrycountry

Mixed Method approach to Question Evaluation and Measurement Construction

Cognitive Test: To understand the ways in which a question performs

Field Test: To understand the extent to which patterns

Objective is to identify:1. Patterns of interpretation2. Calculation processes3. Response error problems4. Interpretive patterns across

subgroups

Establish hypotheses for field test

Cognitive Test: To understand the ways in which a question performs

Field Test: To understand the extent to which patterns exist

Objective is to determine:1. The extent of problematic

patterns2. The extent occurring in

particular subgroups

Test hypotheses from cognitive interviews

Mixed Method approach to Question Evaluation and Measurement Construction

Example: Anxiety

UNESCAP/Washington Group Project

Anxiety Questions

Frequency: How often do you feel worried, nervous or anxious? Daily, weekly, monthly, a few times a year, or never?

Intensity: Thinking about the last time you felt anxious, how would you describe the level of anxiety? Mild, moderate or severe?

Cognitive Interviewing Findings

Interpretations: Aspects of anxiety reported

Stress Workload, deadlines, performances Relationship/family problems Dread/concern about future

Economic security Health concerns

Clinical/diagnosed anxiety Excitement/Positive energy

Additional Probe Questions

Please tell me which of the following statements, if any, describe your feelings.

1. My feelings are caused by the type and amount of work I do.

2. Sometimes the feelings can be so intense that my chest hurts and I have trouble breathing.

3. These are positive feelings that help me to accomplish goals and be productive.

4. The feelings sometimes interfere with my life, and I wish that I did not have them.

5. If I had more money or a better job, I would not have these feelings.

6. Everybody has these feelings; they are a part of life and are normal.

7. I have been told by a medical professional that I have anxiety.

In-Scope, Valid Interpretations

1. Chest Hurts: Sometimes the feelings can be so intense that my chest hurts and I have trouble breathing.

2. Interfere: The feelings sometimes interfere with my life, and I wish that I did not have them

3. Clinical: I have been told by a medical professional that I have anxiety.

Out-of-Scope, Not Valid Interpretations

1. Positive: These are positive feelings that help me to accomplish goals and be productive.

2. Normal: Everybody has these feelings; they are a part of life and are normal.

Anxiety: Frequency

Anxiety: Intensity

Anxiety:Joint distribution of anxiety frequency and intensity

A few times a

year

Monthly Weekly Daily DK/REF

A little 1087 423 328 214 1

Closer to a little

35 25 27 12 0

In between 122 85 95 59 0

Closer to a lot 22 16 39 33 0

A lot 163 86 122 259 0

DK/REF 22 3 3 7 1

Note. Polychoric correlation = .42

Additional Probe Questions

Please tell me which of the following statements, if any, describe your feelings.

1. My feelings are caused by the type and amount of work I do.

2. Sometimes the feelings can be so intense that my chest hurts and I have trouble breathing.

3. These are positive feelings that help me to accomplish goals and be productive.

4. The feelings sometimes interfere with my life, and I wish that I did not have them.

5. If I had more money or a better job, I would not have these feelings.

6. Everybody has these feelings; they are a part of life and are normal.

7. I have been told by a medical professional that I have anxiety.

In-Scope, Valid Interpretations

1. Chest Hurts: Sometimes the feelings can be so intense that my chest hurts and I have trouble breathing.

2. Interfere: The feelings sometimes interfere with my life, and I wish that I did not have them

3. Clinical: I have been told by a medical professional that I have anxiety.

Out-of-Scope, Not Valid Interpretations

1. Positive: These are positive feelings that help me to accomplish goals and be productive.

2. Normal: Everybody has these feelings; they are a part of life and are normal.

Percentage of Reporting Anxiety Characteristics

AnxietyPercent reporting various descriptions of anxiety by country

Kazakhstan

Cambodia

Sri Lank

aMaldiv

esMongoli

aPhilippine

s

Out-of-Scope, Not Valid Interpretations

Positive 50.3% 47.8% 12.6%

51.7% 82.5% 32.4%

Normal 81.5 71.3 75.4 86.7 85.7 81.2

In-Scope, Valid Interpretations

Chest hurts

21.4 72.3 30.9 37.0 50.6 20.3

Interfere

52.2 65.0 85.4 54.8 72.8 33.5

Clinical 11.8 16.8 3.0 28.4 18.6 11.5Note. Chi-square p <.05 for all rows in the table.

“Validity” by countries

Conclusion (A)

1. Question evaluation must be science-based Empirical Evidence Transparent and Systematic analysis With an audit trail, such as Q-Notes to

2. Evaluation studies must be well documented, replicable, and made accessible

3. Data users can improve their analyses with critical use of the data (i.e. questioning the validity of questions and the quality of data)

Conclusion (B)A mixed-method approach for developing

cross-cultural measures:

Can capture interpretations of questionsProvides insight into the extent of those

interpretationsAllows for a comparability examination of

those interpretationsAllows for thoughtful, empirically based

decisions about the measures and the cut-off points