Probability vs non-probability samples. Is Accuracy only ......non-probability based panels are...
Transcript of Probability vs non-probability samples. Is Accuracy only ......non-probability based panels are...
www.pol.gu.se
Probability vs non-probability samples.Is Accuracy only for Probability Samples?
Johan Martinsson, Stefan Dahlberg and Sebastian Lundmark
Department of Political ScienceUniversity of Gothenburg
www.pol.gu.se
The changing world of survey research
1. Mail, face-to-face, and telephone surveys used to dominate sureyresearch
2. Three important changes:
a) lower and lower response rates
b) the introduction of online surveys
c) the increasing use of non-probability samples
3. An large on-going debate concerning the quality of different surveymethods (both modes and samples)
www.pol.gu.se
Question:
1. How do the accuracy of probability based andnon-probability based on-line panels differ?
we compare two probability based panels,and two non-probability based panels
We also compare results to three cross-sectional surveys using different modes:mail, telephone and web
www.pol.gu.se
Surveys from online panelsSurvey
company Mode Sampling methodParticipation
rate
Novus Web panel Prob. based recruitment 59
TNS Sifo Web panel Prob. based recruitment 38
YouGov Web panel Self-recruitment 40
Cint Web panel Self-recruitment (85%) 24
www.pol.gu.se
Cross-sectional surveys with different modes
Surveycompany Mode Sampling method
Responserate
SOM institute Mail Random population sample 53
Detector Telephone Random population sample 51
LORe Web Random population sample 8
www.pol.gu.se
Comparability of surveys
• a set of identical questions were included
• approximatley same period of field work, except for theSOM-institute, which was conducted a few months later
• however, field work length and nr of reminders differ
• we focus on basic demographics and political attitudes
• for demographics, we use census data from StatisticsSweden as benchmark
• target population: 18-70 yrs old, in the Gothenburgregion (west sweden, approx. 1 million inhabitants)
www.pol.gu.se
How do the three cross-sections perform?
www.pol.gu.se
Demographics: average absolute deviation fromStatistics Sweden (unweighted estimates)
Mail(SOM)
Phone(Detector)
Web(LORE)
Average 5 indicators 4.3 3.8 6.9
Sex 5.0 1.0 3.0
Age 4.6 1.8 5.6
Education 5.0 9.0 15.0
Labor market situation 2.5 1.0 2.0
Driving license - 6.0 9.0
www.pol.gu.se
Deviations from Statistics Sweden: age
www.pol.gu.se
Comparing on-line panels: probability basedand non-probability based panels
www.pol.gu.se
We start by examining demographic accuracy once again
www.pol.gu.se
5.7
3.5
2.2
1.3
0 1 2 3 4 5 6
Sifo (Prob)
Novus (Prob)
Cint (Non-P)
YouGov (Non-P)
average abs deviations, demographics(weighted estimates)
TNS Sifo (Prob)
www.pol.gu.se
7
3.7
3.1
2.9
0 1 2 3 4 5 6 7 8
Sifo (Prob)
Novus (Prob)
Cint (Non-P)
YouGov (Non-P)
average abs deviations, demographics(unweighted estimates)
www.pol.gu.se
Novus(Prob)
TNS Sifo(Prob)
YouGov(Non-P)
Cint(Non-P)
Sex 1 1 0 0Age 1.4 1.4 0.2 0.5
Education 6 12 3 1
Labour marketsituation 2 2 9.5
Driving license 7 12 2 0
Demographic deviation, weighted
www.pol.gu.se
Political attitudes
• for attitudes and opinions, there is no true benchmark• however, as second best option, we use the mail survey
as quasi-benchmark• why? :
– well known and high quality survey– excellent sampling frame and high response rates– mode (mail) most similar to web surveys
www.pol.gu.se
We examine three political attitude items:
• Interest in politics (4-point scale)• Trust in politicians (4-point scale)• Attitude towards the introduction of road tolls around
Gothenburg
www.pol.gu.se
7.5
5.9
3.64.6
0123456789
10
Novus (Prob) TNS Sifo (Prob) YouGov (Non-P) Cint (Non-P)
Average absolute attitude deviationfrom a high quality mail survey
www.pol.gu.se
12
43
36
9
20
51
26
3
17
49
31
3
19
46
30
5
17
49
30
4
0
10
20
30
40
50
60
Very interested Fairly interested Not particularlyinterested
Not interested at all
Political interest
www.pol.gu.se
1
39
50
9
3
54
41
23
49
40
8
1
4145
13
1
37
47
15
0
10
20
30
40
50
60
Very high Fairly high Fairly low Very low
Trust in politicians
www.pol.gu.se
62
2118
55
30
15
55
30
15
58
25
17
55
29
16
0
10
20
30
40
50
60
70
Against In favor Don’t know/uncertain
Introduce congestion charges in Gothenburg?
www.som.gu.se
Extremely long field work and representativeness
9 11 13 14 16
23
6056
0
10
20
30
40
50
60
70
80
90
100Percent
16–29 years
Population 16-29 years
Interested in politics
www.pol.gu.se
Conclusions
• surprisingly, the demographic accuracy of thenon-probability based panels are better
• compared to a benchmark mail survey, thenon-probability panels also came closer topolitical attitudes
• in this comparison, we find no evidence thatself-recruited on-line panels have lessaccuracy than probability based on-line panels
www.pol.gu.se
Discussion
• Too much uncertainty about this result
• we would need more demographic indicators than 4-5!
• Sweden also has an extremely high internet coverage
• The self-recruited panels seem to attract more peoplewith low SES, to their advantage
• The probability-based panels are not revealing enoughabout how succesful their recruitment is, are they reallyhigh quality probability based panels?
www.pol.gu.se
Johan Martinsson, Stefan Dahlberg and Sebastian LundmarkDepartment of Political ScienceUniversity of Gothenburg
Thank you for your attention