Do Mechanical Turks Dream of Square Pie Charts?
-
Upload
beliv-workshop -
Category
Documents
-
view
427 -
download
0
description
Transcript of Do Mechanical Turks Dream of Square Pie Charts?
Do Mechanical Turks Dream of Square Pie Charts?
Robert Kosara and Caroline Ziemkiewicz
University vs. Online Studies
University Online
Controlled environment
Easy to recruit Standard incentives
Homogenous users Slow, work-intensive Small samples
Broader demographics Fast, low effort Larger numbers
Environmental control Vote flooding Recruitment and
advertising
2 Caroline Ziemkiewicz
Mechanical Turk Amazon’s micro-payment job market Addresses many issues with online studies!
Prevents vote flooding Manages payment automatically Handles recruitment
3 Caroline Ziemkiewicz
Our experiences Turking studies We’ve done a lot of these by now!
2 years 7 studies Over 550 participants
What have we learned?
4 Caroline Ziemkiewicz
Participants Who are the Turkers?
57.4% female, 42.6% male (in our studies) Contrast with student populations
Mean age of 32, and a broad distribution…
5 Caroline Ziemkiewicz
Participants Generally faster at
tasks than student populations
Some performance differences
Personality
6 Caroline Ziemkiewicz
Designing studies Preventing cheaters
Bonus system allows for accuracy/quality incentives
Building checks into data collection Pre-testing/calibration
Other issues Informed consent Good instructions Feedback Audit trails
7 Caroline Ziemkiewicz
Running studies Participation varies over time of day…
Highest rate during the afternoon + evening EST
8 Caroline Ziemkiewicz
Running Studies Timing affects speed of completion Rate of participation starts high and
decreases over time Placement on front page is crucial!
Longer tasks get fewer participants Turkers like doing studies!
9 Caroline Ziemkiewicz
To Turk or not to Turk Studies that lend themselves well to Turking
Simple interactions and data collection Short performance time Responses that are hard to fake
Studies that don’t Accuracy of self-reported data is crucial Complex tasks Very long task duration
10 Caroline Ziemkiewicz
Thanks!
Let’s discuss!