Multiple Approaches for Evaluating Non-ResponseBias in a Short-Field-Period Survey

Robyn Rapoport, Paul Lavrakas,Eran Ben-Porath & Melissa Herrmann

• The vast majority of surveys have a considerable amount ofnon-response due to refusals and non-contacts

• If the final sample of survey responders differs in non-ignorableway from the non-responders, survey finding will be biased to anon-negligible extent– The past decade has seen a rise in the use of various approaches to

investigate nonresponse bias in sample surveys– It has become a Best Practice to incorporate nonresponse bias studies into

survey designs.

• The vast majority of surveys have a considerable amount ofnon-response due to refusals and non-contacts

• If the final sample of survey responders differs in non-ignorableway from the non-responders, survey finding will be biased to anon-negligible extent– The past decade has seen a rise in the use of various approaches to

investigate nonresponse bias in sample surveys– It has become a Best Practice to incorporate nonresponse bias studies into

survey designs.

Introduction

Prior to the 2013 election, we conducted a survey to assessthe percentage of registered voters who planned to vote inperson who were aware of the new Voter ID laws in anortheastern state and the share of prospective voters whodid not have the necessary documentation to vote or toobtain acceptable voter registration.

Prior to the 2013 election, we conducted a survey to assessthe percentage of registered voters who planned to vote inperson who were aware of the new Voter ID laws in anortheastern state and the share of prospective voters whodid not have the necessary documentation to vote or toobtain acceptable voter registration.

Research Design

Sample Design

• Overlapping dual-frame landline/cell telephonedesign

• More than one third (37%) of the interviews werecompleted via cell phone to ensure that thesample would represent the adult population ofthe STATE

• Overlapping dual-frame landline/cell telephonedesign

• More than one third (37%) of the interviews werecompleted via cell phone to ensure that thesample would represent the adult population ofthe STATE

Questionnaire

• Respondents were asked questions to determineif the respondent was aware of the Voter ID lawand had a Valid ID

• All respondents, even those who were notcitizens or were not registered and not planningto register to vote, were asked demographicquestions for weighting purposes

• Respondents were asked questions to determineif the respondent was aware of the Voter ID lawand had a Valid ID

• All respondents, even those who were notcitizens or were not registered and not planningto register to vote, were asked demographicquestions for weighting purposes

Efforts to Reduce Non-Response

• Interviewers assigned to this study received study-specific training on ways to gain cooperation fromreluctant respondents

• Standard procedures implemented included:– Contacting each non-responsive number up to six times,

varying the time of day and days of week that calls would bemade

– Utilizing specially trained interviewers to re-contacthouseholds where the initial call resulted in a refusal toattempt to convert refusals to completed interviews

• Interviewers assigned to this study received study-specific training on ways to gain cooperation fromreluctant respondents

• Standard procedures implemented included:– Contacting each non-responsive number up to six times,

varying the time of day and days of week that calls would bemade

– Utilizing specially trained interviewers to re-contacthouseholds where the initial call resulted in a refusal toattempt to convert refusals to completed interviews

Non-Response Bias Studies

• Compared data provided by those sampled respondentswho were ‘easy’ to interview to those who were ‘hard’to interview:– Respondents who completed the survey in initial call attempts

to those who completed in later call attempts

– Respondents who completed the survey in initial call attemptsto those who initially refused participation

• Comparisons of responders and non-responders usingauxiliary Census data associated with local zip codes thatwere appended to each landline number

• Analysis of information about refusals collected, in realtime, from the interviewers using a Refusal Report Form(cf. Lavrakas, 2010)

• Compared data provided by those sampled respondentswho were ‘easy’ to interview to those who were ‘hard’to interview:– Respondents who completed the survey in initial call attempts

to those who completed in later call attempts

– Respondents who completed the survey in initial call attemptsto those who initially refused participation

• Comparisons of responders and non-responders usingauxiliary Census data associated with local zip codes thatwere appended to each landline number

• Analysis of information about refusals collected, in realtime, from the interviewers using a Refusal Report Form(cf. Lavrakas, 2010)

Results: Initial Call Attempts vs. Later Call Attempts

• Compared data for key issues by respondents whocompleted the survey in initial call attempts vs.later call attempts

• Outcome measures were: Awareness (unaided and aided) of the legislation requiring

STATE residents to have a valid ID to vote on Election Day Possession of Valid ID

• Compared data for key issues by respondents whocompleted the survey in initial call attempts vs.later call attempts

• Outcome measures were: Awareness (unaided and aided) of the legislation requiring

STATE residents to have a valid ID to vote on Election Day Possession of Valid ID

Data show some differences by call attempt for aidedawareness; small sample size and non-linear relationship donot suggest presence of non-response bias

77.1%

22.9%

79.0%

21.0%

80.9%

19.1%

78.2%

21.3%

Knows about ID law Doesn't know about

ID law

Call Attempts 1 2 3 4+

Unaided(n=1,258)Unaided(n=1,258)

60.9%

37.5%

67.2%

31.1%

50.0% 50.0%

82.9%

14.6%

Knows about ID law Doesn't know about

ID law

Call Attempts 1 2 3 4+

Aided(n=272)Aided

(n=272)

Denotes significance at the 0.05 level

Data do not point to differences by call attempt forpossession of Valid ID or by those who are both awareof the law and have a Valid ID

96.6%

3.4%

96.6%

3.4%

97.3%

2.7%

98.3%

1.7%

Has valid ID No valid ID

Call Attempts 1 2 3 4+

75.2%

24.8%

76.2%

23.8%

78.6%

21.4%

77.4%

22.6%

Aware of law and have

ID

Not aware of law or

do not have valid ID

Call Attempts 1 2 3 4+

(n=1,213) (n=38)(n=959) (n=296)

Results: Initial Refusers vs. Non-Refusers

• Respondents who initially refused to cooperate werere-contacted by specially trained interviewers

• Of those re-contacted, 144 agreed to complete thesurvey

• Among these initial refusers, 87% (n=125) wereregistered to vote and planned to vote in person andwere therefore asked substantive questions in survey

• Respondents who initially refused to cooperate werere-contacted by specially trained interviewers

• Of those re-contacted, 144 agreed to complete thesurvey

• Among these initial refusers, 87% (n=125) wereregistered to vote and planned to vote in person andwere therefore asked substantive questions in survey

There are no statistically significant differences betweeninitial and non-refusers regarding unaided or aidedawareness of the law

78.3%

21.5%

81.6%

18.4%

Knows about ID law Doesn't know

about ID law

Non-Refuser Initial Refuser

64.3%

34.4%

56.5%

39.1%

Knows about ID law Doesn't know

about ID law

Non-Refuser Initial Refuser

Unaided(n=1,251)Unaided(n=1,251)

Aided(n=267)Aided

(n=267)

Both initial refusers and non-refusers are equally likely to have avalid form of identification; data do not suggest differences bythose who are both aware of the law and have a Valid ID

97.0%

3.0%

76.4%

23.6%

96.8%

3.2%

80.0%

20.0%

Have valid ID No valid ID Aware of law and

has ID

Not aware of law or

does not have valid

ID

Non-Refuser Initial Refuser

(n=1,213) (n=38)

(n=959)

(n=291)

Results: Zip Code Level Differences

• Each landline phone number was matched to its mostlikely zip code

• Compared census-level statistics for the local area inwhich people lived to determine whether those whocompleted the survey differed from those who did notcomplete on area-level statistics

• Each landline phone number was matched to its mostlikely zip code

• Compared census-level statistics for the local area inwhich people lived to determine whether those whocompleted the survey differed from those who did notcomplete on area-level statistics

Of the 32 factors included in the model, only the share of certainethnic groups in the exchange was significantly associated withthe propensity to complete the survey

WorkingNumbers

Exp(B) p≤ Known Households Exp(B) p≤

% Hispanic inExchange

.989 .021 % Hispanic .989 .020

% AA inExchange

.996 .047 % AA NS -

% Asian inExchange

.964 .021 % Asian NS -

Results: Refusal Report Form (RRF)

• Information gathered via an RRF can be used tohelp the refusal conversion process and alsoprovides paradata to allow researchers to studythe nature of refusal-related nonresponse in asurvey

• It also may be beneficial in studying nonresponsebias

• Information gathered via an RRF can be used tohelp the refusal conversion process and alsoprovides paradata to allow researchers to studythe nature of refusal-related nonresponse in asurvey

• It also may be beneficial in studying nonresponsebias

For this study, the RFF was not helpful in ourinvestigation into nonresponse bias

• We believe that the RFF was not effective primarilybecause this pool of interviewers had never used such aform before and generally seemed to lack confidence inusing it

• We now realize that the interviewers should havereceived specific training (including practice) in the useof the form; usage of the RFF would have require moreup-front preparation time than what was available forthis survey

• We believe that the RFF was not effective primarilybecause this pool of interviewers had never used such aform before and generally seemed to lack confidence inusing it

• We now realize that the interviewers should havereceived specific training (including practice) in the useof the form; usage of the RFF would have require moreup-front preparation time than what was available forthis survey

Conclusions and Implications• As response rates continue to decline, it is increasingly

important to incorporate non-response bias studies intothe research design to provide assurance that the surveydata collected are valid and meaningful

• It is possible to embed multiple non-response biasstudies into surveys that have relatively short fieldperiods

• These studies did not show evidence of non-responsebias for either awareness of the Voter ID law orpossession of a Valid ID, the key statistics for this survey

• More work can and should be done to investigate thepresence of non-response biases

• As response rates continue to decline, it is increasinglyimportant to incorporate non-response bias studies intothe research design to provide assurance that the surveydata collected are valid and meaningful

• It is possible to embed multiple non-response biasstudies into surveys that have relatively short fieldperiods

• These studies did not show evidence of non-responsebias for either awareness of the Voter ID law orpossession of a Valid ID, the key statistics for this survey

• More work can and should be done to investigate thepresence of non-response biases

Thank You

Robyn RapoportVice President – Policy Research, SSRS

rrapoport@ssrs.com53 West Baltimore Pike

Media, Pennsylvania 19063www.ssrs.com@ssrs_solutions