GOR 2018 | 2nd March 2018

Effects of mobile devices on data quality in web-based surveys

2

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Device Ownership

90% 43% 58%

2017 2017 2016

Laufende Wirtschaftsrechnung, Statistisches Bundesamt

3

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Mobile Devices

>15‘‘ 7-12‘‘ <6‘‘

External Keyboard

+ Mouse

Internal Keyboard

+ Touch

Internal Keyboard

+ Touch

Cable or WiFiWiFi or mobile

network

WiFi or mobile

network

4

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Difficulties when Assessing the Effects

SELECTION EFFECTS

MEASUREMENT EFFECTS

5

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Propensity Score Matching

AGE

GENDER

HOUSEHOLD INCOME

EDUCATION

EMPLOYMENT STATUS

EXPERIENCE WITH SURVEYS

AFFINITY FOR TECHNOLOGY

INTERNET USAGE

EXPERIENCE WITH DEVICE

Comparison of “similar” respondents within treatment and control group

QUASI EXPERIMENT

6

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Data

Gesis Panel

10th wave (August 2015)

Not optimized for mobile

7

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Indicators for Data Quality

LENGTH OF OPEN

ANSWERS

PRIMACY

EFFECTS

SURVEY

DURATION

8

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Results

LENGTH OF OPEN ANSWERS

9

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

* p < 0,10 ** p < 0.05 *** p < 0.01

Results

- 23** Characters - 9 Characters + 3 Characters

LENGTH OF OPEN ANSWERS

10

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Results

PRIMACY EFFECTS

11

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

+ 22 PP***more 1./2. answers

+ 1 PPmore 1./2. answers

+/- 0 PPmore 1./2. answers

* p < 0,10 ** p < 0.05 *** p < 0.01

Results

PRIMACY EFFECTS

12

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Results

SURVEY DURATION

13

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

* p < 0,10 ** p < 0.05 *** p < 0.01

Results

+ 6.5** minutes + 4.6* minutes + 4.7** minutes

SURVEY DURATION

14

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Lower Evaluation of the Study

Higher Item Nonresponse

Shorter Open Answers

Higher Primacy Effects

Stronger Left-Alignment

Longer Duration

Conclusion

15

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Responsive

survey design

Item nonresponse:

Make answers

mandatory

Multiple choice

questions:

Randomize answer

options

Exclusion of

mobile devices or

asking the

respondents to

switch

Possible Solutions

Optimization for

and testing on a

variety of devices

16

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Data

30 SKOPOS surveys

2017

Optimized for mobile devices

17

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Participation by Device

Single survey

76%

24%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

18

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Data Quality

Closed Questions

(Average std

in item set)

Open-Ended Questions

(Number of characters written)

0,860,96

60,5352,51

19

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Completion Rate

Single survey

82%

77%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

20

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

When do respondents drop out? (Average seconds)

363 sec.333 sec.

21

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

But is survey length and drop-out rate correlated?

R² = 0,057

0%

25%

50%

75%

100%

0 5 10 15 20

Average interview duration (minutes)

Com

plet

ion

Rat

e R² = 0,0657

0%

25%

50%

75%

100%

0 5 10 15 20 25

Single survey

Average interview duration (minutes)

22

GOR 2018 | Effects of mobile devices on data quality in web-based surveys

Conclusion

Survey

Duration

(Sweetspot: 5 min.)

Mobile ExperienceResponsive Design

23

GOR 2018 | Learning From All Answers | March 1, 2018

Further questions? Let‘s talk!

Thank You For Your Attention!

Patrick Schlickmann

Project Manager

patrick.schlickmann@skopos.de

Sebastian Schmidt

Director Research & Development

sebastian.schmidt@skopos.de