striving to earn more - Susumu S · 2018. 11. 13. · HIT Scraper Tools 0 20 40 60 80 100...
Transcript of striving to earn more - Susumu S · 2018. 11. 13. · HIT Scraper Tools 0 20 40 60 80 100...
Striving to Earn More: A Survey of Work Strategies and Tool Use Among Crowd Workers
Toni Kaplan*, Susumu Saito*, Kotaro Hara, and Jeffrey P. Bigham
(* Equal contribution)
Earning a decent wage is difficult for crowd workers on Amazon Mechanical Turk• Most workers are not paid well [Martin et al., 2014]
• Workers’ median hourly wage is ~$2/h, while average requester pays $11+/h [Hara et al., 2018]
Supporting workers for a better wage is a crucial challenge
What is needed to improve wage?(i.e., increasing !"#$% &$'(&) '&*$')
!"#$% #+,& -.&(# $- $*"'/&' )
1. Minimizing spent time 2. Maximizing “pay per work time”
[Hanrahan et al., 2015]
(Given )
…But not everyone can do this*Task listing page on Amazon Mechanical Turk*
✘ Requires time to find tasks [Chilton et al. 2010]
✘ Likely to waste working time [McInnis et al. 2016]
- Too small reward- Broken task (= unable to submit)- Answer rejection after submission
Work Strategies
• Website forums and tools/scripts [Mason and Suri, 2012]
→ Some workers are successful, but others are still not
! No formal research on tool types and theirprevalence per workers’ earnings
Our future project: Support by automation using Artificial Intelligence (AI)?→ Modeling successful workers’ behavior on their work
can possibly give all workers good capabilities
<
Research Questions
a) What are workers’ current strategies/tools/information for improving wage?
b) What are design considerations for future tool development for supporting crowd workers?
Survey
• Period … January 23, 2018 – January 31, 2018
• Subjects … 360 US-based workers on Amazon Mechanical Turk• 40/400 spam answers are filtered out• 100 each to workers who has completed {100 | 1,000 | 5,000 | 10,000} HITs
• Task … 67 survey questions (~10-30 minutes) for $3.50• General demographic information and income as workers• Workers’ working tips• Workers’ tools/forums usage• Sentiment towards AI …
Analysis
• Worker categorization by their reported total earnings in 2017
High-earning Extremes … Top 10%
High-earners … Top 50%
Low-earners … Bottom 50%
Outline
1. Tools and strategies for workers
2. Sentiment towards automation with AI
3. Discussion: How can we assist workers better?
Outline
1. Tools and strategies for workers
2. Sentiment towards automation with AI
3. Discussion: How can we assist workers better?
0
20
40
60
80
Always Most of the time About half the time Occasionally Never
High-earning extremes High-earners Low-earners
Website forums
[%]
Q: How often do you browse or post in Mechanical Turk related websites?
Website forums
Q: Which of the following Mechanical Turk related websites do you regularly read / browse / reference?
• MTurkgrind• MTurk Forum• Turker Nation• HIT Notifier
• MTurk Crowd• Turker Hub• [Reddit] AMT• [Reddit] HITsWorthTurkingFor• [Reddit] hNOTwtf
Website forums
0
20
40
60
80
100
MTurk Crowd Turker Hub [Reddit]mturk
[Reddit]HWTF
…
[%]
• Relatively new and active• Various thread categories
(e.g., “Daily Work”, “MTurk Help & Resources” …)
• Active• Mainly Q&A / chatting• Single thread category
0
50
100
Extremes High-earners Low-earners
Q: Are you using any browser extensions or scripts to improveyour work experience/pay on Mechanical Turk?
Tools
[%]
Tools
Q: Which, if any, of the following extensions and scripts do you regularly use while working on Mechanical Turk?
• HIT Scraper
• Turkmaster
• MTurk Engine
• Turkopticon
• MTurk Suite
• Panda Crazy
HIT Scraper
Tools
0
20
40
60
80
100
Turkopticon MTurk Suite Panda Crazy
Turkopticon [Irani et al., 2013]
MTurk Suite
Panda Crazy:• Userscript with interface
for automated “PandA”ing of
prespecified HIT batches
HIT Scraper:• Auto-refreshing task search
with extra filters
(Turkopticon ratings,
customizable blocklist…)
Outline
1. Tools and strategies for workers
2. Sentiment towards automation with AI
3. Discussion: How can we assist workers better?
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
StronglyAgree
SomewhatAgree
Neither Agreenor Disagree
SomewhatDisagree
StronglyDisagree
“I would use a tool that automates some of the work in a HIT, which then lets me verify if the work was done correctly.”
Why NOT agree?
Why would you NOT use it?
• 78 workers expressed concerns• The role of a human in Human Intelligence Tasks (18)
• “The whole purpose of a HIT is to complete a Human Intelligence Task, which by definition is a task that cannot or should not be automated.”
• Mistrust in the quality of AI output (17)• “I don’t trust it, it would add more time to go back and check to see if it was right”
• Violating the AMT terms of service (12)• Violation of their personal ethics (9)• Unfairness to requesters (12)
Outline
1. Tools and strategies for workers
2. Sentiment towards automation with AI
3. Discussion: How can we assist workers better?
Is there room for improvement?STATUS QUO: “Sharing” information among workers
Microtasks are doneall by workers
Tools that assistsearching tasks(e.g., Turkopticon)
(Choose tasks)
Tools that assistaccepting tasks(e.g., Panda Crazy)
(Popular tasks)
Is there room for improvement?
2. Tools that supportdoing microtasks
POSSIBLE FUTURE: Collecting data for automation with AI
Workers do microtasks with the support
1. Tools that automatesearch & accepting microtasks
(Popular tasks)
(Similar tasks)
Proposal 1: Automatic microtask search
Find and reserve recommendable tasks as soon as they are posted
SupervisedML algorithms
Output:- Hourly wage- How possible to be
approved?- How long will they
be available?
?Newly posted
microtask batch
Input:
Training dataFeature: microtask HTMLLabel: completion time,
approval/rejection…
Worker preferences- Microtask type- Work time- Reward target …
Auto-reservemicrotasks
Proposal 2: Iterative answer suggestion
Augmenting work by automating user action• Opening links as a task starts (e.g., survey tasks)• Copy & pasting specified keywords in microtasks (e.g., external search tasks)
Suggesting answers for repeated or commonly asked questions• “Input your AMT worker ID” etc.
|Answer suggestions:
AXBCDEFG01234
(Repeatedlyinputting
worker ID)
(Repeatedlyinputting
worker ID)
SupervisedML algorithms?
(Training data) (Output)
Conclusion
a) What are workers’ current strategies/tools/information for improving wage efficiency?→ Use active forums and tools that evaluate requesters/tasks and
reserve task batches
b) What are design considerations for future tool development for supporting crowd workers?→ Use collected data for automation with machine learning
Striving to Earn More: A Survey of Work Strategies and Tool Use Among Crowd WorkersToni Kaplan*, Susumu Saito*, Kotaro Hara, and Jeffrey P. Bigham(* Equal contribution)
Questions?
0
20
40
60
80
Always Most of the time About half the time Occasionally Never
Extremes High-earners Low-earners
Website forums
[%]
Q: Do you browse or post in any Mechanical Turk related websites?
Website forums
0
20
40
60
80
100
MTurkCrowd
Turker Hub [Reddit]mturk
[Reddit]HWTF
[Reddit]hNOTwtf
Mturkgrind MTurkForum
TurkerNation
HIT Notifier Other
[%]
Website forums
• MTurk Crowd and Turker Hub• Relatively new and rapidly growing• Has many discussion threads
• Subreddits• Popular among most workers, but less attractive to high-earning extremes• Dunno why?
• Other forums• Used to be active but not anymore.
• HIT Notifier• Still new and seems pretty easy to use, but dunno why it’s not popular?
Tools
0
20
40
60
80
100
Turkopticon MTurkSuite
Tampermonkey/Greasemonkey
Panda Crazy
(Userscript-related)(No userscript-related)
Tools
• Turkopticon and MTurk Suite• Just a chrome extension and intuitive, which makes it very easy to use for everyone• Used mainly for seeing requesters’ reputation
• Greasemonkey, Tampermonkey• A base extension tool for userscript
• Panda Crazy• Not very intuitive but has high functionality w/ helper scripts, once getting used
• HIT Scraper• Intuitive, but not always good for high-earners who prefers native AMT page
• Turkmaster• Often recognized as a substitute of Panda Crazy. Seems less useful
• MTurk Engine• A new script being developed recently, that combines Panda Crazy and HIT Scraper• Maybe not widely known yet?
High-earners had more access to
• Masters Qualification• Access to more profitable HITs• More opportunity, higher reward
• “PANDA” technique (“Preview and Accept”)• Technique for task batch reservation• More opportunity, less task search time
0
50
100
Extremes High-earners Low-earners
0
50
100[%]
[%]