Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels?...
Transcript of Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels?...
![Page 1: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/1.jpg)
Do Response Rates Matter inOnline Panels?
Representativity at Different Levels ofCumulative Response Rates
Johan MartinssonUniversity of Gothenburgwww.lore.gu.se
![Page 2: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/2.jpg)
About response rates
• RR used to be the main indicator of data quality
• Perhaps today more questionable (e.g. Groves &Peytcheva 2008)
• Moving to online surveys and panels, even more so
• Are participation rates for online panels meaningful?
![Page 3: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/3.jpg)
www.mod.gu.sewww.mod.gu.se
Yeager et al. 2011, pp. 731
![Page 4: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/4.jpg)
www.mod.gu.sewww.mod.gu.se
0
5
10
15
20
25
30
35
40
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
No
ofsu
rvey
s
Participation rate (%)
Participation rates in 163 surveys from theCitizen Panel at the University of Gothenburg
The most important predictor is the demographiccomposition of the invited sample
![Page 5: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/5.jpg)
www.mod.gu.se
Moving to more controlled comparison: theprobability based part of the Citizen Panel
• The emerging indicator is now Cumulative ResponseRate for probability based panels
• CRR = Recr.Rate * Profile rate * Participation rate
• Both recruitment rates and CRRs are usually low
• And it is not clear how important they are
![Page 6: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/6.jpg)
www.mod.gu.se
Briefly about the Citizen Panel and LORE
• Started in 2010 with support from the University ofGothenburg
• The Citizen Panel has approx. 9,000 members fromprobability based recruitments from population samples
• And approximately 57,000 members from opt-inrecruitments
• You can read more at www.lore.gu.se
![Page 7: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/7.jpg)
www.mod.gu.se
Response rates in web panels
• In 2012 LORE conducted a large probability recruitmentwith an experimental design
• This results in variation in recruitment rates
![Page 8: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/8.jpg)
www.mod.gu.sewww.mod.gu.se
Recruitment rates in 9 different treatment groups
![Page 9: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/9.jpg)
www.mod.gu.se
Questions
• What happens to response rates and CRR over time ina panel?
• How are they related to accuracy/representativity?
![Page 10: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/10.jpg)
www.mod.gu.se
How to examine representativeness/accuracy
• We calculate average absolute deviations from a goodbenchmark
• 6 demographic indicators– Gender, age, country of birth, marital status,
education, labour market situation– These benchmarks come from Statistics Sweden– Four of these also register data in the sample
![Page 11: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/11.jpg)
www.mod.gu.se
How to examine representativeness/accuracy
• 2 political indicators (quasi benchmarks)
– Interest in politics (from the SOM Institute survey,mail, n=6000)
– Vote intention (Statistics Sweden, RDD, n=9000)
![Page 12: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/12.jpg)
www.mod.gu.se
![Page 13: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/13.jpg)
www.mod.gu.se
1. What happens to CRR over time in a panel?
![Page 14: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/14.jpg)
www.mod.gu.se
![Page 15: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/15.jpg)
www.mod.gu.se
• Initial span 8.4 pct
• After two yrs 2.7 pct
• Higher RR => higherattrition
• Incentives recruitment=> higher attrition
![Page 16: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/16.jpg)
www.mod.gu.se
2. How are RR and CRR related toaccuracy/representativity?
• First, we examine the accuracy at the recruitment
• Next, accuracy over time in the panel
![Page 17: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/17.jpg)
www.mod.gu.se
2. How are RR and CRR related toaccuracy/representativity?
• At recruitment we have 11 treatment groups withdifferent recruitment rates between 6 and 21 percent
• We examine the average the AAD for the 8 indicators
![Page 18: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/18.jpg)
www.mod.gu.se
All 8 indicators
r=-.61
![Page 19: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/19.jpg)
www.mod.gu.se
r=-.30
All 8 indicators
![Page 20: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/20.jpg)
www.mod.gu.se
Vote intention (8 parties)
r=-.44
![Page 21: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/21.jpg)
www.mod.gu.se
Vote intention (8 parties)
r=-.14
![Page 22: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/22.jpg)
www.mod.gu.se
2. How are RR and CRR related toaccuracy/representativity?
• We observe a weak correlation at the recruitment stage
• But what about later on? For that purpose we have asmaller set of groups that allow strict comparisons
• We examine representativeness 2 years later (8 panelwaves later)
![Page 23: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/23.jpg)
www.mod.gu.sewww.mod.gu.se
Standard postcards: all 8 indicators
Standard cardno reminder
Standard card1 reminder
CRR AAD CRR AADWave 0 5.8 9.8Wave 1 4.7 7.3Wave 8 2.7 4.8
![Page 24: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/24.jpg)
www.mod.gu.sewww.mod.gu.se
Standard postcards: all 8 indicators
Standard cardno reminder
Standard card1 reminder
CRR AAD CRR AADWave 0 5.8 7.2 9.8 6.9Wave 1 4.7 7.5 7.3 6.6Wave 8 2.7 8.4 4.8 8.2
![Page 25: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/25.jpg)
www.mod.gu.sewww.mod.gu.se
Standard postcards: 2 political indicators
Standard cardno reminder
Standard card1 reminder
CRR AAD CRR AADWave 0 5.8 6.6 9.8 6.0Wave 1 4.7 7.6 7.3 6.4Wave 8 2.7 10.1 4.8 8.5
![Page 26: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/26.jpg)
www.mod.gu.sewww.mod.gu.se
Incentive postcards: all 8 indicators
Incentive cardno reminder
Incentive card1 reminder
CRR AAD CRR AADWave 0 9.3 14.2Wave 1 6.4 9.5Wave 8 3.6 5.2
![Page 27: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/27.jpg)
www.mod.gu.sewww.mod.gu.se
Incentive postcards: all 8 indicators
Incentive cardno reminder
Incentive card1 reminder
CRR AAD CRR AADWave 0 9.3 6.5 14.2 6.9Wave 1 6.4 7.7 9.5 7.4Wave 8 3.6 9.2 5.2 8.6
![Page 28: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/28.jpg)
www.mod.gu.sewww.mod.gu.se
Incentive postcards: 2 political indicators
Incentive cardno reminder
Incentive card1 reminder
CRR AAD CRR AADWave 0 9.3 4.3 14.2 5.9Wave 1 6.4 4.8 9.5 6.4Wave 8 3.6 7.0 5.2 9.7
![Page 29: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/29.jpg)
www.mod.gu.sewww.mod.gu.se
Did you notice something peculiar?
Standard cardno reminder
Standard card1 reminder
CRR AAD CRR AADWave 0 5.8 7.2 9.8 6.9Wave 1 4.7 7.5 7.3 6.6Wave 8 2.7 8.4 4.8 8.2
![Page 30: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/30.jpg)
www.mod.gu.sewww.mod.gu.se
Incentive cardno reminder
Incentive card1 reminder
CRR AAD CRR AADWave 0 9.3 6.5 14.2 6.9Wave 1 6.4 7.7 9.5 7.4Wave 8 3.6 9.2 5.2 8.6
Did you notice something peculiar?
![Page 31: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/31.jpg)
www.mod.gu.se
Summing up
• High initial recruitment rates deteriorate substantiallyover time and differences diminish
• Higher cumulative response rates consistently yieldlower average errors ...– ... when comparing over time within a recruitment
cohort– ... and when comparing between cohorts at same
wave / panel age• But panel attrition seems to play a part independent of
cumulative response rates (because not at random)• Overall, correlations btw CRR and accuracy seem low• Caveats: small samples for accuracy, unweighted data,
limited variation in recruitment rates and CRRs
![Page 32: Do Response Rates Matter in Online Panels?...Do Response Rates Matter in Online Panels? Representativity at Different Levels of Cumulative Response Rates Johan Martinsson University](https://reader030.fdocuments.net/reader030/viewer/2022040514/5e6ad05b55b1ed3f7c6f347c/html5/thumbnails/32.jpg)
www.mod.gu.se
Read more at
www.lore.gu.se
Laboratory of Opinion Research(LORE)