Analysis of Factors Related to Nonresponse for Landline and

Cell Phone Surveys in China

Yan Jiang

School of Statistics, Renmin University of China

Visiting Professor, Iowa State University, Department of Statistics and Center for Survey Statistics and Methodology

ITSEWITSEW20102010

04/10/232

Outline

Introduction Survey Design Nonresponse Rates Call Attempt Analysis Interviewer Analysis Conclusions

04/10/233

Introduction

Surveys in China High nonresponse rates

Low contact rates (RDD Sampling)High refusal rates

Change of coverage Steady increase in cell phone frame Decrease in landline phone frame

since 2006

04/10/234

Amount of telephone users

0

10000

20000

30000

40000

50000

60000

70000

80000

90000

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

cell phone users

landline telephone households

(Ten thousands)

landline telephone users

2009

Cellphone Users:

747.38 million

Landline Users:697.02 million

Overlap: unknown

Total :1335 million

Survey Objectives

To evaluate the utility of selected methods (e.g., increased call effort) in telephone surveys of adults in China

To identify factors related to nonresponse rates (e.g., interviewer voice quality, call effort)

To study how the effect of these factors on nonresponse vary by sample frame (i.e., landline, cell)

04/10/235

Survey Design

Population: Adults in China (over 18 years old) Two-phase Sampling Design

Phase 1: Dual frame sample of phone numbers to sample adults

Phase 2: Sample of nonrespondents

Data Collection Mode: CATI Phase 2: Modified methods

04/10/236

Survey Design in First Phase Sample Size: 17,162 phone numbers

Cell phone: 9678 Landline: 7484

Stratified RDD Sampling Strata: Provinces Allocation: proportional to Province HH pop.

Standard call methods Up to 3 call attempts – usually 2 Ring for about 20 secs

04/10/237

Survey Design in Second Phase Population: nonrespondents in first phase

Sample size: 1549 Strata: province, frame source, noncontact

and refusal numbers Effort to improve the response rate (both

cell phone and landline survey) Up to 15 call attempts were made to contact Wait until at least one minute in each dial

process

04/10/238

Nonresponse Rate in First Phase Survey

Landline Survey Cell phone Survey

Number Percent Number Percent

Total 190,680 100% 169,116 100%

Non-Eligible 149,224 78.3% 108,213 64%

Eligible 41,456 21.7% 60,903 36%

Response 7,484 18.0% 9,678 15.9%

Non-Response 33,972 81.9%81.9% 51,225 84.1%84.1%

Non-Contact 23,446 56.6% 36,716 60.3%

Refusal 10,526 25.4% 14,509 23.8%

Nonresponse Rate in Second Phase Survey

landline cellphone

sample size rate sample size Rate

response 763 43.6%43.6% 786 45.0%45.0%

nonresponse 985 56.4% 961 55.0%

non-contact 443 25.4% 467 26.7%

refuse 542 31.0% 495 28.3%

04/10/2311

Number of call attempts in refusal sample (stratum) of Phase 2

Cumulated Refusal Conversion Rate of landline survey is higher than cell phone survey when the number of attempts is less than 5.

Cumulated Refusal Conversion Rate increases rapidly up to about 6 call attempts.

LandlineCell phone

Number of call attempts

04/10/2312

logistic model for Refusal Conversion Dep. Variable=1, refusal unit convert to respondent

landline survey cell phone survey

Variable B Sig. Variable B Sig.

Gender -.014 .604 Gender -.196 .000

Young -.791 .000 Young -1.056 .000

Middle-aged -.767 .000 Middle-aged -1.068 .000

Education -.189 .000 Education .187 .000

Income -.066 .000 Income -.090 .000

City -.066 .024 City .156 .000

East 1.236 .000 East .941 .000

Middle -.367 .000 Middle .984 .000

Numberof call attempts

.146 .000Numberof call attempts

.097 .000

Constant -1.621 .000 Constant -2.241 .000

The relationship between above factors and refusal conversion rates differs for the landline survey and the cell phone survey.

04/10/2313

Number of call attempts in non-contact sample (stratum) of Phase 2

Cumulated Phase 2 Response Rate of landline survey is lower than cell phone survey in non-contact sample.

LandlineCell phone

Number of call attempts

04/10/2314

Distribution of the number seconds until the phone is answered

100.0080.0060.0040.0020.000.00

等待时间

200

150

100

50

0

成功手机频数

Mean = 32.5089Std. Dev. = 10.2653N = 786

120.00100.0080.0060.0040.0020.000.00

等待时间

250

200

150

100

50

0

成功固定电话频数

Mean = 19.5464Std. Dev. = 10.30387N = 765

On average,10 additional seconds per interview are required for the phone to be answered in the cell phone survey relative to the landline survey.

Cell PhoneCell Phone

Land lineLand line

04/10/2315

Phase 1 Interviewer Analysis Cooperation rate distribution of interviewers

0.700.600.500.400.300.20

Cooperation Rate

20

15

10

5

0

Fre

qu

en

cy

Mean = 0.375

Std. Dev. = 0.083

N = 108

Phase 1 Interviewer Analysis

Which interviewer characteristics affect cooperation rates?

Interviewer characteristicsSkill levelVocal quality

Control for respondent characteristicsGender, education, career, income,

location of residence, sample frame

04/10/2316

Phase 1 Interviewer Analysis

Hierarchical Generalized Linear Model Level 1 model (respondent level)

Level 2 model : (interviewer level)

q qijqjj

ij

ij X)p

plog( 01

qjs sjqsqqj W 0

~ (0, )qj qqN

qijX

sjW

i : respondent; j: interviewer; :refusal rate : variables of respondent : variables of interviewer

ijp

04/10/2318

Variable Definitions

outcome of interview

y =1, refusal; y =0, response

respondent

X

gender=1, male; gender=0, female

edu1: low education, edu2: midium education

career1: student, career2: manager, career3: clerk, career4: self-employee

income>=0

survey =1, landline ; survey =0, cell phone

grade: =1, large city; grade=0, other

InterviewerW

W1: interview skill of interviewer

W2: vocal characteristic of interviewer

( | )ij ij sp prob y W

Results

Both the interviewers’ speaking skill and natural

vocal characteristics have significant impacts

on refusal rates.

Both the demographic characteristics of respondents and interviewers have

significant impacts on refusal rates.

Null model

Level 1 model

model

Level 2 model

model Fixed effect Coefficients SE Coefficients SE Coefficients SE

respondent level

intercept，00 0.545** 0.060 0.702** 0.105 0.691** 0.093

career1，10 -1.974** 0.409 -2.056** 0.426

career2，20 -2.275** 0.440 -2.194** 0.426

career4，30 -2.767** 0.463 -2.928** 0.488

income，40 0.00016* 0.00007 0.00016* 0.00007

grade，50 1.266** 0.205 1.307** 0.214

interviewer level

W1，01 -0.28** 0.059

W2，02 -0.146* 0.059

career4W2, 32 1.136** 0.134

Survey type does not have significant effects on refusal rates in the

model

Respondent Interviewer

Conclusions1. Dual frame sampling is necessary for reducing coverage bias

for telephone surveys in China. The ineligible (e.g., non-working, business) rate in cell phone surveys appears to be lower than in landline surveys in China.

2. High non-response rates can be reduced by effective voice training of interviewers, more call attempts, and extending waiting time before giving up on the call.

3. The demographic characteristics and behaviors of cell phone users and landline telephone users are different, which may have implications for nonresponse bias.

4. Interviewers should wait longer for the respondent to answer the phone in the cell phone survey relative to the landline survey.

5. Refusals occur more quickly in the cell phone survey than in the landline survey.