Does Sample Source Matter? [Part 1]
-
Upload
vcuniversity -
Category
Documents
-
view
924 -
download
2
description
Transcript of Does Sample Source Matter? [Part 1]
![Page 1: Does Sample Source Matter? [Part 1]](https://reader035.fdocuments.net/reader035/viewer/2022081907/54813135b4795979578b4867/html5/thumbnails/1.jpg)
Does the Sample Source Matter? A comparison of some leading sample sources
October 3, 2011
![Page 2: Does Sample Source Matter? [Part 1]](https://reader035.fdocuments.net/reader035/viewer/2022081907/54813135b4795979578b4867/html5/thumbnails/2.jpg)
Sample is the soil and water of research. The quality of it determines the value of what you produce.
![Page 3: Does Sample Source Matter? [Part 1]](https://reader035.fdocuments.net/reader035/viewer/2022081907/54813135b4795979578b4867/html5/thumbnails/3.jpg)
We compared SBA to four other sample
sources that VC regularly uses
We conducted a five minute survey Nov 4 to 7, 2011 with a sample of about 500 people each, using sample from Vision Critical’s Springboard America, and four of the leading sample providers that we have anonymised as “Panel B”, “ Panel C”, “Panel D” and “River E”. The questionnaire included items from the US 2010 General Social Survey, as well as a questions about snack consumption and a concept test for a snack food. We counted the number of statistically significant differences between the panels (95% CI)
![Page 4: Does Sample Source Matter? [Part 1]](https://reader035.fdocuments.net/reader035/viewer/2022081907/54813135b4795979578b4867/html5/thumbnails/4.jpg)
Sample Quality: Dropout rate during the survey
4%
8% 10%
7%
16%
0%
5%
10%
15%
20%
SBA Panel B River C Panel D River E
Dropouts as a percent of all who started the survey
![Page 5: Does Sample Source Matter? [Part 1]](https://reader035.fdocuments.net/reader035/viewer/2022081907/54813135b4795979578b4867/html5/thumbnails/5.jpg)
Data quality: straight-lining
0.5%
6.5%
7.9%
3.9%
7.8%
0%
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
SBA Panel B River C Panel D River E
% who gave the same answer for all 13 trust ratings
![Page 6: Does Sample Source Matter? [Part 1]](https://reader035.fdocuments.net/reader035/viewer/2022081907/54813135b4795979578b4867/html5/thumbnails/6.jpg)
Data quality: use of “don’t know”
Average percent who choose the option "don’t know", "can't choose", "not sure", "prefer not to say".
4%
6% 6%
4%
6%
0%
1%
2%
3%
4%
5%
6%
7%
8%
SBA Panel B River C Panel D River E
Average % "Don't Know"
![Page 7: Does Sample Source Matter? [Part 1]](https://reader035.fdocuments.net/reader035/viewer/2022081907/54813135b4795979578b4867/html5/thumbnails/7.jpg)
Differences between panels in how people
answered all questions
5%
11%
4%
6% 7%
-1%
1%
3%
5%
7%
9%
11%
13%
15%
SBA Lightspeed SSI Research Now Fulcrum
Percent of times panel differed from the others
Panel C Panel D River
Count of the number of statistically significant differences (95% CI) as a percent of the total number of differences possible
Expected by chance
Panel B
![Page 8: Does Sample Source Matter? [Part 1]](https://reader035.fdocuments.net/reader035/viewer/2022081907/54813135b4795979578b4867/html5/thumbnails/8.jpg)
Differences between panels in how people
answered snack purchase questions
Count of the number of statistically significant differences (95% CI) as a percent of the total number of differences possible
2%
13%
2%
6% 6%
-1%
1%
3%
5%
7%
9%
11%
13%
15%
SBA Panel B River C Panel D River E
Purchase behaviour: Percent of times panel differed from the others
They were less likely to buy snacks
![Page 9: Does Sample Source Matter? [Part 1]](https://reader035.fdocuments.net/reader035/viewer/2022081907/54813135b4795979578b4867/html5/thumbnails/9.jpg)
Differences between panels in how people
answered concept test questions
Count of the number of statistically significant differences (95% CI) as a percent of the total number of differences possible
11%
26%
6%
16%
13%
0%
5%
10%
15%
20%
25%
30%
SBA Lightspeed SSI Research Now Fulcrum
Concept Test: Percent of times panel differed from the others
Panel B Panel C Panel D River
![Page 10: Does Sample Source Matter? [Part 1]](https://reader035.fdocuments.net/reader035/viewer/2022081907/54813135b4795979578b4867/html5/thumbnails/10.jpg)
Differences between panels in how people
answered the purchase intent question
Count of the number of statistically significant differences (95% CI) as a percent of the total number of differences possible
13% 6%
9% 12% 8%
21%
15%
23%
28%
21%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
SBA Panel B River C Panel D River E
Definitely will Probably will
![Page 11: Does Sample Source Matter? [Part 1]](https://reader035.fdocuments.net/reader035/viewer/2022081907/54813135b4795979578b4867/html5/thumbnails/11.jpg)
Conclusions
Sample matters. A lot. Springboard America has fewer drop-outs, notably better data quality, and is “right down the middle” when it comes to comparison to the other panels on the findings, unlike some panels which show real variation—particularly on the concept test.