RTI International is a trade name of Research Triangle Institutewww.rti.org
Effect of Questionnaire Structure on Nonresponse and Measurement Error: Sequential vs. Grouped Placement of Filter Questions
Lisa R. Carley-Baxter, RTI InternationalAndy Peytchev, RTI International
Michele Lynberg Black, CDC
5/18/2010
2
www.rti.org
Overview
• Introduction• Sequential vs Grouped Structure• Hypotheses• Study Design• Study Specific Filter and Follow-up Questions• Results• Discussion
5/18/2010
3
www.rti.org
Introduction
• Structure of a questionnaire can impact survey estimates through multiple types of measurement error
• Researchers aim to structure questionnaires in way that respondents’ memories are organized
• Survey instruments often include follow-up questions that depend on response to a previous question (filter question)
5/18/2010
4
www.rti.org
Sequential versus Grouped Structure
• Sequential: – Filter Question 1 -> Follow-up Questions– Filter Question 2 -> Follow-up Questions
• Grouped: – Filter Question 1...Filter Question X– Follow-up Questions to Filter 1 (if endorsed)– Follow-up Questions to Filter 2 (if endorsed)
• Sequential structure is assumed to help recall, reduce measurement error, and possibly decrease item nonresponse
5/18/2010
5
www.rti.org
Literature
• Suggests that respondents may underreport in order to avoid additional questions– Kessler et al (1998), Duan et al (2007)– Respondents reported lower mental distress or mental
health service use when a sequential design was used– Both studies were very long face-to-face interviews– Both studies targeted minority groups that differ from the
general population on measures related to cognition and willingness to conduct surveys
• Research Question: Does a sequential design increase reporting in a 30 minute telephone survey of the general population?
5/18/2010
6
www.rti.org
Hypotheses
• Interview length will be longer in the sequential structure than in the grouped structure.
• Higher number of breakoffs, number of behaviors reported, and item nonresponse in the sequential structure compared to the grouped structure.
• Interviewers will prefer the grouped structure over the sequential structure.
5/18/2010
7
www.rti.org
Study Design
• National Intimate Partner and Sexual Violence Surveillance System (NISVSS)– National RDD survey of US adults 18+– Collects sensitive information on psychological aggression,
coercive control and entrapment, physical violence, stalking, and sexual violence (ever, past 3 yrs, past 12 months)
– Stratified dual-frame RDD sample design that included both landline and cell phone numbers
– Oversampled males during respondent selection– Respondent selection:
• 2 adults: Modified Rizzo method• 3+ adults: Most recent birthday method
5/18/2010
8
www.rti.org
Study Design (cont’d)
• 6 week pre-test conducted Nov – Dec, 2009– Included a 2 week nonresponse phase
• Lead letters sent to all landline cases for which an address was obtained
• Cases randomly assigned to either the sequential or grouped structure
• 784 total completed interviews– 495 sequential structure– 289 grouped structure
• AAPOR Response Rate #1 = 12.9%
5/18/2010
9
www.rti.org
NISVSS Filter and Follow-up Questions
• Filter: How many of your romantic or sexual partners have ever slapped you?– Follow-up: Please tell me the initials of the (first, etc) person
who slapped you.– Follow-up: How many times did (initials) ever slap you?
• Filter: How many of your romantic or sexual partners have ever pushed or shoved you?– Follow-up: Please tell me the initials of the (first, etc) person
who pushed or shoved you.– Follow-up How many times did (initials ever) push or shove
you?
5/18/2010
10
www.rti.org
Mean Length of Interview
Sequential 32.3Grouped 32.6
Sequential 30.3Grouped 31.0
Sequential 31.4Grouped 31.8
Females
Males
Total
5/18/2010
11
www.rti.org
Mean Breakoff Rates
Sequential 19.9Grouped 23.9
Sequential 16.7Grouped 19.8
Sequential 18.5Grouped 22.0
Females
Males
Total
5/18/2010
12
www.rti.org
Location of Breakoffs
Sequential Grouped Sequential Grouped Sequential GroupedPrior to Victimization Sections 24.2 40.4 15.9 44.1 20.9 42During Victimization Sections 63.6 51.1 70.5 55.9 66.4 53.1
After Victimization Sections 12.1 8.5 13.6 0 12.7 4.9**p<.01
Females Males** Total**
5/18/2010
13
www.rti.org
Mean Number of Behaviors Reported
Sequential 6.3Grouped 5.9
Sequential 3.5Grouped 4.6
Sequential 5.0Grouped 5.1
Males
Total
Females
5/18/2010
14
www.rti.org
Item Nonresponse
Sequential 0.5Grouped 0.2
Sequential 0.6Grouped 0.4
Sequential 0.5Grouped 0.4
Females
Males
Total
5/18/2010
15
www.rti.org
Regression Analysis
• Ran 6 Poisson regression equations by sex – Lifetime victimization– 5 of which used individual violence types as dependent
variable • Psychological aggression• Coercive control and entrapment • Physical violence • Stalking • Sexual violence
– 1 of which used any reported behavior experienced as the dependent variable
5/18/2010
16
www.rti.org
Regression Results
• Females:– Sequential structure was a significant predictor of increased
reporting:• Coercive control and entrapment• Physical violence• Sexual violence
• Males:– Sequential structure was a significant predictor of decreased
reporting:• Coercive control and entrapment• Sexual violence• Stalking
• Both: Age, age squared, and lower levels of education had stronger and more consistent effects on reports of experiencing the 5 individual violence types and any behavior
5/18/2010
17
www.rti.org
Interviewer Feedback
• Held meetings with interviewers throughout the data collection period
• Conducted an interviewer debriefing towards the end of the data collection period
• Interviewers consistently reported that they and respondents disliked the grouped structure– Main reason: Confusing to have to go back and ask follow-
up questions about specific perpetrators after the initial reports
• Reported sequential structure was easier to administer and clearer for respondents
5/18/2010
18
www.rti.org
Summary of Results
• Very few significant difference found between structures in the bivariate analysis– Only significant difference was where cases broke off: higher
breakoffs in victimization section for sequential structure
• Regression results paint a different picture– Sequential structure significant predictor of 3 violence types
each for both sexes (though different violent types, and opposite impact on reporting)
5/18/2010
19
www.rti.org
Discussion
• Most interesting finding:– Higher reports for females in sequential structure– Higher reports for males in grouped structure
• Interaction of gender and structure is of both theoretical and practical importance– Suggests that optimal questionnaire structure depends on
the sample members and topics of the questions
5/18/2010
20
www.rti.org
Discussion (cont’d)
• Sequential structure:– Higher breakoff rates within a section of detailed behavior
questions– But also overall reports for some of those questions may be
higher for the key population group of interest
• Important for studies where researchers are concerned about underreporting of experiences– Example: intimate partner and sexual violence
• Use of different structures may interfere with comparison across studies
Top Related