Ipek BilgenDavid SterrettMichael J. Stern

Exploring Nonresponse andCoverage in a Web Study

AAPOR 2015

Ipek Bilgen, David Sterrett, and Michael J. Stern

• Multi-mode surveys: Increasingly include a webcomponent (concurrently or subsequently)

• Web-surveying became a viable data collection mode– 86% of all U.S. adults now go online

– Compared to 50% in 2000

– The current national broadband adoption rate = 70% (Pew 2014)

• Rise in web enabled mobile devices– The national smartphone adoption rate = 55% (Pew 2014)

• Though increase in RRs to web surveys has notaccelerated as it was expected.

Background

Source: http://www.pewinternet.org 2

• Coverage: Lack of internet access, sufficientinternet speed/proficiency

• Nonresponse: Lack of time/interest/willingness/trust(despite access)

Why people do not respond to web?

3

• Research gap: Disentangle coverage from non-response inweb surveys

• Studies conducted a decade ago looking into this issueindicate that non-coverage is more concerning than non-response for web surveys (Couper et al. 2007, Lee 2006)

• Significant increase in internet use in the last decade, whichlikely means an increase in web coverage.

– Given the dynamic nature of this issue, there is a possibilitythat web coverage is less of an issue now.

Introduction

4

• Disentangle web-specific non-coverage and non-response.

• Examine reasons for nonresponse to web surveys

• Rationale: Understand the reasons of web nonresponseamong individuals with internet access

• Prior to data collection: Tailor data collection strategies todrive more households to the less-expensive web mode

• Post data collection: Tailor post data collection adjustmentstrategies to obtain less biased estimates

Study Aim

5

• Is non-response becoming more of a concernthan non-coverage?

• Which demographic groups are more likely to be notcovered in internet surveys?

• (Controlling for internet access/use) Whichdemographic groups are more reluctant to participatein internet surveys?

– What are the potential reasons for non-response?

Research Questions

6

Healthy Illinois Study (HIL)

• Questions on behaviors and opinions related tothe searching for and transferring of online orelectronic health records

• Sample Size (Web Survey) = 4,000• Sequential Multi-mode Study

• WEB, SAQ, CATI• Match physical addresses to phone numbers (landlines).

Data & Methods (I)

7

• Sequential Multi-mode: Followed up with non-respondents to the HIL WEB survey

• NR Sample: where a phone number (landline) is matched to aphysical address (n=1795)

• Step 1: Mailed out a self-administrated questionnaire (SAQ)booklet (including $2 incentive)

• Step 2: Administered the questionnaire via telephoneinterviews (CATI) for four weeks approximately three weeksafter the SAQ mailings.

• In both modes, we asked sample members a series of questions:– Their electronic device ownership; Mobile internet access; Whether

respondents remember receiving the web invitation letter or e-mail;Their reasons for not responding if they remember such invitations

Data & Methods (II)

8

MODE Samplesize

# ofresponse

Adjusted response rates(%)

WEB(Initial mode) 4,000 487 12.2%

SAQ(NR follow-up) 1,795 352 19.6%

CATI(NR follow-up) 1,455 168 11.5%

TOTAL 1007 37.7% (AAPOR RR2)

Response Rates (per mode)

9

• Based on three questions:• Number of devices own that could access internet

– Do you personally own…[Desktop computer; Laptop computer; Cell phone;Blackberry or iPhone or other handheld device that is also a cell phone]?

– Coded from 0 (own no devices) to 4.(own all four devices).

• Mobile access– Do you have access to the Internet through a mobile device (such as a

smart phone, personal digital assistant (PDA), iPhone, or Blackberry?”– Coded from 0 (no) to 1 (yes).

• Frequency access Internet– How often do you access the Internet? Daily, Several times a week,

Several times a month, Several times a year or less often, or Never?Coded from 0 (never) to 4 (daily).

• Internet Access/Use Scale (0-9) = number of devices own (0-4) +mobile access (0-1)+ frequency access internet (0-4)

Internet Access/Use Scale

10

Internet Access/Use Scale Distribution

0

20

40

60

80

100

120

140

160

0 1 2 3 4 5 6 7 8 9

Num

ber o

f res

pond

ents

Internet Access/Use score

Web

SAQ and CATI

Non-Coverage (NC):12% Web and 30% ofSAQ/CATI haveinternet access/usescore of 0-5

Non-Response (NR):88% Web and 70% ofSAQ/CATI haveinternet access/usescore of 6 or more

11

0%

20%

40%

60%

80%

100%

18-30 31-40 41-50 51-60 61+

% o

f Res

pond

ents

Respondent Age

Non-Coverage(Score = 0-5)

Non-Response(Score = 6-9)

NR > NC among middle aged respondents

Please note that,* indicates p < 0.05; ** indicates p< 0.001; *** indicates p< 0.0001

NC vs. NR**

12

MODE = SAQ and CATI Follow-up

0%

20%

40%

60%

80%

100%

HS andbelow

Associate's Bachelor's Graduate

% o

f Res

pond

ents

Respondent Education

Non-Coverage(Score = 0-5)

Non-Response(Score = 6-9)

NR > NC among respondents with more than HS education

Please note that,* indicates p < 0.05; ** indicates p< 0.001; *** indicates p< 0.0001 13

NC vs. NR***

MODE = SAQ and CATI Follow-up

0%

20%

40%

60%

80%

100%

% o

f Res

pond

ents

Respondent Income

Non-Coverage(Score = 0-5)

Non-Response(Score = 6-9)

NR > NC among higher income respondents

14

NC vs. NR***

MODE = SAQ and CATI Follow-up

0%

20%

40%

60%

80%

100%

Excellent Good Fair Poor

% o

fRes

pond

ents

Respondent Health

MODE = SAQ and CATI Follow-up

Non-Coverage(Score = 0-5)

Non-Response(Score = 6-9)

NR > NC among healthier respondents (self-report)

15

NC vs. NR***

Asked if they rememberedreceiving an invitation (26%)

Reasons for not responding

0%10%20%30%40%50%60%70%80%90%

100%

Don't haveaccess to

web/internet

Did nothave time

Wanted tocomplete but

forgot

Did not findinvitation

email/letterto be

informative

% re

spon

ded

YE

S SAQCATI

16

• The majority of the SAQ and CATI respondents couldpotentially be reached via web since they report:

• Having daily access to internet (~60%)• Having mobile internet access (~40%)• Personally own a smart phone (~39%)

• Can be potentially reached via less expensive web mode• Demographic break-down of potential Web NRs• Learn where need to put more emphasis, creativity• Tailor our data collection strategies to drive more households to

the web survey– Decrease burden to enter URL into browser– Eliminate procrastination: Early-bird incentives in web– Increase trust (some groups are reluctant to respond via web

due to security reasons)

Key Findings & Discussion

17

• The SAQ and CATI respondents (with internet access) whoremember receiving the mail/invitation letters

• Majority indicated did not have time (~ 36%)• Wanted to complete but forgot (~ 37%)• Did not find invitation email/letter informative (~15%)

• However, only 26% of these respondentsremember receiving the mail/invitation letters forthe web survey (potential issues with junk mailand/or address & phone match)

Key Findings & Discussion

18

• Time lag between initial survey and follow up• May cause recall issues

– High % of missing item NR for the questions related to receiving theweb invitation letter or e-mail

• Potential Social desirability bias (CATI)• Qs related to why respondents did not complete the web survey

– High % of “did not have time” responses in CATI.

• External validity• The study only looks at residents of one state, which could limit

the ability to make generalizations about the population.– Can be tested by comparing the sample demographics to census

figures and by comparing responses to the substantive questions tothe benchmark survey results.

Limitations

19

Thank You!

Questions?

bilgen-ipek@norc.org

0%10%20%30%40%50%60%70%80%90%

100%

Yes No Don'thaveemail

DK Yes No DK

EMAIL LETTER

SAQ

CATI

Received web invitation letter or e-mail about the study?

21

0%10%20%30%40%50%60%70%80%90%

100%

Desktop Laptop CellPhone

SmartPhone

% re

spon

ded

YE

S

Do you personally own?

WEB

SAQ

CATI

Electronic device ownership

22

0%10%20%30%40%50%60%70%80%90%

100%

Never Severaltimes a year

(or less)

Severaltimes amonth

Severaltimes aweek

Daily

How often do you access the internet?

WEBSAQCATI

Internet Usage

23

0%10%20%30%40%50%60%70%80%90%

100%

Yes No

Do you have access to theInternet through a mobile device?

WEBSAQCATI

Mobile internet access

24