Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner...
29
Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 [email protected]
Transcript of Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner...
Survey Design 201
Getting from “Doing a survey” to “Doing a survey right.”
Andy ZehnerOffice of AssessmentYoung Hall, Rm 521
Basic tips
1. Don’t ask what you already know2. Avoid biased or leading questions3. Make choice options:
1. exhaustive2. exclusive3. balanced
4. Avoid double-barrelled questions5. Don’t assume knowledge of terms
6. Avoid jargon7. Be brief 8. Be specific9. Do/Don’t include a “neutral” option10. Avoid complex negative constructions11. Order questions strategically12. Save controversial questions for last
Spot the flaw
1. “Rate the quality and variety of the food at Wiley Dining Court”
2. “CCO Career Services are:”• Excellent Outstanding Very good Good Average
3. “Have you participated in any flipped courses during the past year?”• Yes No
4. Were the clerical staff who served you in [office] courteous?1. Yes2. Neutral3. No
5. What portion of the material did you master by the end of the training session?• All Most Some Very little None
New considerations
• Vague quantifiers• Individual response style• Dunning-Kruger Effect
Dunning-Kruger Effect
Kruger & Dunning, Journal of Personality and Social Psychology, 1999
What evidence of Dunning Kruger Effect looks like
Very poor Poor Fair Good Very good Excellent0%
10%
20%
30%
40%
50%
60%
Freshmen 1Freshmen 2Seniors 1Seniors 2
2014 Purdue SERU survey
Vague quantifiers
Vague quantifiers
How often have you…
Had a class where the professor knew your name
Never
Rarely
Occasio
nally
Somewhat o
ftenOften
Very often
0%
5%
10%
15%
20%
25%
30%
Communicated with an instructor by email/in person
Never
Rarely
Occasio
nally
Somewhat o
ftenOften
Very often
0%
5%
10%
15%
20%
25%
30%
2014 SERU Survey, Purdue data
Vague quantifiers
• “Always” almost always means always• “Never” almost always means never• Any other vague quantifier is . . . vague• Vague quantifiers convey relative position only• Meaning varies from one person to another• Meaning varies from one question to another• Grammatical opposites aren’t perceived equally
“Sometimes”
• “asking questions in class” 2 times per week• “working with other students outside of class” 1 time per month• “making a class presentation” 2 times per
semester• “participating in a community-based project” 1 time per year
How Often is Often Revisited: the Meaning and Linearity of Vague Quantifiers Used on the National Survey of Student Engagement -- Laird, et al., Indiana University, 2008
Vague quantifiers of behavioral frequency, Marincic, 2011
“Stay away from using words like “often,” “usually,” “generally,” etc. Each person’s thought process is different and some people may infer a different meaning.”
-- Smart Survey Design, Survey Monkey, 2011
Retrieved 3/11/2015
Consider using idiosyncratic terms!
2013 Purdue Student Satisfaction Survey
Response style
Moderate & Extreme Response
Bursar’s drop-box
Very unim
portant/d
issati
sfied
Unimporta
nt/diss
atisfi
ed
Neither
Importa
nt/Sati
sfied
Very im
portant/s
atisfi
ed0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Importance Satisfaction
Registering for classes
Very unim
portant/d
issati
sfied
Unimporta
nt/diss
atisfi
ed
Neither
Importa
nt/Sati
sfied
Very im
portant/s
atisfi
ed0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Importance Satisfaction
2010 Purdue Student Importance & Satisfaction Survey
Quotes from focus groups, Purdue, 2012-2013
“I feel like here in the US people use stronger
language, like ‘Oh, this is so wonderful.’ But for me,
I’m not so willing to say it.”
“Americans use a lot of hyperbole: ‘It’s super, it’s
wonderful!’ That is not language we use where
I come from.”
Factors that determine acquiescence
• Individualism -.445 *• Power distance .372 *• Extraversion .446 **• Uncertainty avoidance .533 *
Response styles in cross-national survey research: a 26-country study, Harzing, et al, 2006
Response styles by nationality
30 35 40 45 50 55 60 655
10
15
20
25
30
Acquiescence
Extr
eme
Resp
onse
Sty
le
Japan
India
Germany
Holland
Mexico
Turkey
China
Some words don’t translate well
“good” v. 良い / いい
“normal” v. нормальный
Does America spend too little, about the right amount, or too much money on…
Too little About right Too much0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
"Welfare" "Assistance to the poor"
General Social Survey, 2014
Again: Does America spend too little, about the right amount, or too much money on…
Liberal respondents
Too little About right Too much0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
"Welfare" "Assistance to the poor"
Conservative respondents
Too little About right Too much0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
"Welfare" "Assistance to the poor"
Order of choice options matters
Purdue’s favorite pizza:
Hot Box / Mad Mushroom / Villa@PMU / Domino’s
Domino’s / Villa@PMU / Mad Mushroom / Hot Box
Biasing Effects of Scale-Checking Style in Response to a Likert Scale, American Statistical Assn, Friedman, et al., 1994
Numbers on the scale matter, as much as the words do
0 1 2 3 4 56 7 8 9 10
Not at allExtremely
Successfulsuccessful
-5 -4 -3 -2 -1 01 2 3 4 5
Not at allExtremely
Successfulsuccessful
Rating the Rating Scales, Journal of Marketing Management, Friedman & Amoo, 1999
You can get reliable results with a moderate sample size or response rate
Stronglydisagree
Disagree Neutral Agree Stronglyagree
0%5%
10%15%20%25%30%35%40%
True distribution Sample of 5000
Stronglydisagree
Disagree Neutral Agree Stronglyagree
0%5%
10%15%20%25%30%35%40%
True distribution Sample of 1000
Stronglydisagree
Disagree Neutral Agree Stronglyagree
0%5%
10%15%20%25%30%35%40%
True distribution Sample of 500
Stronglydisagree
Disagree Neutral Agree Stronglyagree
0%5%
10%15%20%25%30%35%40%
True distribution Sample of 50
After the shot went off, my
training took over.
Resources and links:
A good overall lesson in 106 slides
Survey Monkey’s guide
Cross–national response styles
Rating the Rating Scales
Vague quantifiers: How Often is Often?
Vague quantifiers: Marincic dissertation
More about importance/performance analysis
Tex Grebner’s video (language)
