www.pol.gu.se

Probability vs non-probability samples.Is Accuracy only for Probability Samples?

Johan Martinsson, Stefan Dahlberg and Sebastian Lundmark

Department of Political ScienceUniversity of Gothenburg

www.pol.gu.se

The changing world of survey research

1. Mail, face-to-face, and telephone surveys used to dominate sureyresearch

2. Three important changes:

a) lower and lower response rates

b) the introduction of online surveys

c) the increasing use of non-probability samples

3. An large on-going debate concerning the quality of different surveymethods (both modes and samples)

www.pol.gu.se

Question:

1. How do the accuracy of probability based andnon-probability based on-line panels differ?

we compare two probability based panels,and two non-probability based panels

We also compare results to three cross-sectional surveys using different modes:mail, telephone and web

www.pol.gu.se

Surveys from online panelsSurvey

company Mode Sampling methodParticipation

rate

Novus Web panel Prob. based recruitment 59

TNS Sifo Web panel Prob. based recruitment 38

YouGov Web panel Self-recruitment 40

Cint Web panel Self-recruitment (85%) 24

www.pol.gu.se

Cross-sectional surveys with different modes

Surveycompany Mode Sampling method

Responserate

SOM institute Mail Random population sample 53

Detector Telephone Random population sample 51

LORe Web Random population sample 8

www.pol.gu.se

Comparability of surveys

• a set of identical questions were included

• approximatley same period of field work, except for theSOM-institute, which was conducted a few months later

• however, field work length and nr of reminders differ

• we focus on basic demographics and political attitudes

• for demographics, we use census data from StatisticsSweden as benchmark

• target population: 18-70 yrs old, in the Gothenburgregion (west sweden, approx. 1 million inhabitants)

www.pol.gu.se

How do the three cross-sections perform?

www.pol.gu.se

Demographics: average absolute deviation fromStatistics Sweden (unweighted estimates)

Mail(SOM)

Phone(Detector)

Web(LORE)

Average 5 indicators 4.3 3.8 6.9

Sex 5.0 1.0 3.0

Age 4.6 1.8 5.6

Education 5.0 9.0 15.0

Labor market situation 2.5 1.0 2.0

Driving license - 6.0 9.0

www.pol.gu.se

Deviations from Statistics Sweden: age

www.pol.gu.se

Comparing on-line panels: probability basedand non-probability based panels

www.pol.gu.se

We start by examining demographic accuracy once again

www.pol.gu.se

5.7

3.5

2.2

1.3

0 1 2 3 4 5 6

Sifo (Prob)

Novus (Prob)

Cint (Non-P)

YouGov (Non-P)

average abs deviations, demographics(weighted estimates)

TNS Sifo (Prob)

www.pol.gu.se

7

3.7

3.1

2.9

0 1 2 3 4 5 6 7 8

Sifo (Prob)

Novus (Prob)

Cint (Non-P)

YouGov (Non-P)

average abs deviations, demographics(unweighted estimates)

www.pol.gu.se

Novus(Prob)

TNS Sifo(Prob)

YouGov(Non-P)

Cint(Non-P)

Sex 1 1 0 0Age 1.4 1.4 0.2 0.5

Education 6 12 3 1

Labour marketsituation 2 2 9.5

Driving license 7 12 2 0

Demographic deviation, weighted

www.pol.gu.se

Political attitudes

• for attitudes and opinions, there is no true benchmark• however, as second best option, we use the mail survey

as quasi-benchmark• why? :

– well known and high quality survey– excellent sampling frame and high response rates– mode (mail) most similar to web surveys

www.pol.gu.se

We examine three political attitude items:

• Interest in politics (4-point scale)• Trust in politicians (4-point scale)• Attitude towards the introduction of road tolls around

Gothenburg

www.pol.gu.se

7.5

5.9

3.64.6

0123456789

10

Novus (Prob) TNS Sifo (Prob) YouGov (Non-P) Cint (Non-P)

Average absolute attitude deviationfrom a high quality mail survey

www.pol.gu.se

12

43

36

9

20

51

26

3

17

49

31

3

19

46

30

5

17

49

30

4

0

10

20

30

40

50

60

Very interested Fairly interested Not particularlyinterested

Not interested at all

Political interest

www.pol.gu.se

1

39

50

9

3

54

41

23

49

40

8

1

4145

13

1

37

47

15

0

10

20

30

40

50

60

Very high Fairly high Fairly low Very low

Trust in politicians

www.pol.gu.se

62

2118

55

30

15

55

30

15

58

25

17

55

29

16

0

10

20

30

40

50

60

70

Against In favor Don’t know/uncertain

Introduce congestion charges in Gothenburg?

www.som.gu.se

Extremely long field work and representativeness

9 11 13 14 16

23

6056

0

10

20

30

40

50

60

70

80

90

100Percent

16–29 years

Population 16-29 years

Interested in politics

www.pol.gu.se

Conclusions

• surprisingly, the demographic accuracy of thenon-probability based panels are better

• compared to a benchmark mail survey, thenon-probability panels also came closer topolitical attitudes

• in this comparison, we find no evidence thatself-recruited on-line panels have lessaccuracy than probability based on-line panels

www.pol.gu.se

Discussion

• Too much uncertainty about this result

• we would need more demographic indicators than 4-5!

• Sweden also has an extremely high internet coverage

• The self-recruited panels seem to attract more peoplewith low SES, to their advantage

• The probability-based panels are not revealing enoughabout how succesful their recruitment is, are they reallyhigh quality probability based panels?

www.pol.gu.se

Johan Martinsson, Stefan Dahlberg and Sebastian LundmarkDepartment of Political ScienceUniversity of Gothenburg

Thank you for your attention