Crowdsourcing with Amazon Mechanical Turk
-
Upload
tasha-arnold -
Category
Documents
-
view
57 -
download
4
description
Transcript of Crowdsourcing with Amazon Mechanical Turk
![Page 1: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/1.jpg)
Crowdsourcing withAmazon Mechanical Turk
Alex Quinn and Tom Yeh
HCIL SymposiumMay 25, 2011
![Page 2: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/2.jpg)
Interface options
• Easy web-based tool– http://requester.mturk.com
• REST and SOAP APIs (HTTP)• Wrapper SDKs: Java, .NET, Perl, PHP, Ruby• Command line tool• TurKit (JavaScript)• Boto (Python)• CrowdLib (Python)
![Page 3: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/3.jpg)
Posting HITs – external
Your URL in an IFRAME
Your form must route results to Amazon.<form method="post" action="https://mturk.com/...">
![Page 4: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/4.jpg)
Posting HITs – standard
QuestionForm XML
![Page 5: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/5.jpg)
Standard vs. External HITs
Standard• Amazon hosts everything
• Content: HTML, Flash, applets
• Answers: Standard controls
• No JavaScript / CSS
• More secure
• Easier to get started
External• Requires your own web server
• Content: anything!
• Answers: Your CGI form fields
• Inner scrolling (potentially)
• More flexible
• Easier to do fancy things
![Page 6: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/6.jpg)
API Documentation
![Page 7: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/7.jpg)
Question XML<QuestionForm xmlns="..."> <Question> <QuestionIdentifier>food_name</QuestionIdentifier> <IsRequired>true</IsRequired> <QuestionContent> <Text>What food is Maine known for?</Text> </QuestionContent> <AnswerSpecification> <FreeTextAnswer></FreeTextAnswer> </AnswerSpecification> </Question></QuestionForm>
![Page 8: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/8.jpg)
Example - TurKitvar params = {
title : "Find food for a state",desc : "Given a US state find a food it is known for",question : questionXML,reward : 0.01,maxAssignments : 3
};
var hit = mturk.createHIT(params);
hit = mturk.getHIT(hit,true);
for( var j=0; j<hit.assignments.length; j++ ) { var foodName = hit.assignments[j].answer.food_name; print(stateName + " : " + foodName);}
<QuestionForm xmlns="..."> <Question> <QuestionIdentifier>food_name</QuestionIdentifier> <IsRequired>true</IsRequired> <QuestionContent> <Text>What food is Maine known for?</Text> </QuestionContent> <AnswerSpecification> <FreeTextAnswer></FreeTextAnswer> </AnswerSpecification> </Question></QuestionForm>
![Page 9: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/9.jpg)
Example - CrowdLib
import crowdlib as cl, crowdlib_settings
hit_type = cl.create_hit_type("Find food for a state", "Given a US state find a food it is known for", 0.01)
fields = [cl.text_field("What food is Maine known for?")]
hit = hit_type.create_hit(fields, max_assignments=3)
![Page 10: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/10.jpg)
Example – Command Line• Many little files!– state_foods.question– state_foods.input– state_foods.properties– state_foods_start.bat– state_foods_get_results.bat– state_foods_review_results.bat
• Creates even more…– state_foods.success– state_foods.results
![Page 11: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/11.jpg)
Example – Command Line• Many little files!– state_foods.question question XML– state_foods.input parameters (i.e. states)– state_foods.properties title, description, etc.– state_foods_start.bat calls loadHITs.bat– state_foods_get_results.bat fetch results– state_foods_review_results.bat approve/reject
• Creates even more…– state_foods.success contains the HIT IDs– state_foods.results all result details
![Page 12: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/12.jpg)
Example – Command Line• Many little files!– state_foods.question– state_foods.input– state_foods.properties– state_foods_start.bat– state_foods_get_results.bat– state_foods_review_results.bat
• Creates even more…– state_foods.success– state_foods.results
<QuestionForm xmlns="..."> <Question> <QuestionIdentifier> food_name </QuestionIdentifier> <IsRequired>true</IsRequired> <QuestionContent> <Text>
What food is Maine known for? </Text> </QuestionContent> <AnswerSpecification> <FreeTextAnswer/> </AnswerSpecification> </Question></QuestionForm>
![Page 13: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/13.jpg)
Example – Command Line• Many little files!– state_foods.question– state_foods.input– state_foods.properties– state_foods_start.bat– state_foods_get_results.bat– state_foods_review_results.bat
• Creates even more…– state_foods.success– state_foods.results
stateAlabamaAlaskaArizonaArkansasCaliforniaColoradoConnecticutDelawareDistrict of ColumbiaFloridaGeorgiaHawaiiIdahoIllinoisIndianaIowaKansasKentuckyLouisianaMaine...
![Page 14: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/14.jpg)
Example – Command Line• Many little files!– state_foods.question– state_foods.input– state_foods.properties– state_foods_start.bat– state_foods_get_results.bat– state_foods_review_results.bat
• Creates even more…– state_foods.success– state_foods.results
title:Find food for a state
description:Given a US state find a food it is known for
keywords:culture, food, search
reward:0.02
assignments:1
annotation:${state}
# time limit: 30 minutesassignmentduration:1800
# expires in: 3 dayshitlifetime:259200
# auto-pay after: 2 daysautoapprovaldelay:1296000
![Page 15: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/15.jpg)
Example – Command Line• Many little files!– state_foods.question– state_foods.input– state_foods.properties– state_foods_start.bat– state_foods_get_results.bat– state_foods_review_results.bat
• Creates even more…– state_foods.success– state_foods.results
pushd bin
call loadHITs %1 %2 %3 %4 %5 ^ -label ..\sf\state_foods ^ -input ..\sf\state_foods.input ^ -question ..\sf\state_foods.question ^ -properties ..\sf\state_foods.properties
popd
![Page 16: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/16.jpg)
Example – Command Line• Many little files!– state_foods.question– state_foods.input– state_foods.properties– state_foods_start.bat– state_foods_get_results.bat– state_foods_review_results.bat
• Creates even more…– state_foods.success– state_foods.results
pushd ..\bin
call getResults %1 %2 %3 %4 %5 ^ -successfile ..\sf\state_foods.success -outputfile ..\sf\state_foods.results
popd
![Page 17: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/17.jpg)
Example – Command Line• Many little files!– state_foods.question– state_foods.input– state_foods.properties– state_foods_start.bat– state_foods_get_results.bat– state_foods_review_results.bat
• Creates even more…– state_foods.success– state_foods.results
pushd ..\..\bin
call reviewResults %1 %2 %3 %4 %5 ^ -resultsfile ..\sf\state_foods.results
popd
![Page 18: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/18.jpg)
Example – Command Line• Many little files!– state_foods.question– state_foods.input– state_foods.properties– state_foods_start.bat– state_foods_get_results.bat– state_foods_review_results.bat
• Creates even more…– state_foods.success– state_foods.results
hitid hittypeid11MFAR2I9GUBPPX3H302KECWDGG45C 1MQHAMAJ67VR562077Y5NLM6WNOIO81N2WE274D6AYYDS5F6ENYRH1NR7JAO 1MQHAMAJ67VR562077Y5NLM6WNOIO81KBVYNQOT8Y85Z3T3H6YTS1CV3596T 1MQHAMAJ67VR562077Y5NLM6WNOIO81NPDB0OPE4D7JGXHYWSMAPKKCVAELG 1MQHAMAJ67VR562077Y5NLM6WNOIO81HNBWMEOVNP9NL2O6SYS8WM19JGNIZ 1MQHAMAJ67VR562077Y5NLM6WNOIO8
![Page 19: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/19.jpg)
Example – Command Line• Many little files!– state_foods.question– state_foods.input– state_foods.properties– state_foods_start.bat– state_foods_get_results.bat– state_foods_review_results.bat
• Creates even more…– state_foods.success– state_foods.results
"hitid" "hittypeid" "title" "description" "keywords" "reward" "creationtime" "assignments" "numavailable" "numpending" "numcomplete" "hitstatus" "reviewstatus" "annotation" "assignmentduration" "autoapprovaldelay" "hitlifetime" "viewhit" "assignmentid" "workerid" "assignmentstatus" "autoapprovaltime" "assignmentaccepttime" "assignmentsubmittime" "assignmentapprovaltime" "assignmentrejecttime" "deadline" "feedback" "reject" "Answer.answer"
"11MFAR2I9GUBPPX3H302KECWDGG45C" "1MQHAMAJ67VR562077Y5NLM6WNOIO8" "Find food for a state" "Given a US state find a food it is known for" "US culture, food, search" "$0.02" "Mon Dec 06 21:50:25 EST 2010" "1" "0"
"0" "0" "Reviewable" "NotReviewed" "Maine" "3600" "1296000" "Thu Dec 09 21:50:25 EST 2010" "http://requester.mturk.com/mturk/manageHIT?HITId=11MFAR2I9GUBPPX3H302KECWDGG45C" "1SPKY563G67J32JCIGFCFF0CVAKGF0" "A1JYO3377GTS1O" "Submitted" "Tue Dec 21 21:52:56 EST 2010" "Mon Dec 06 21:51:26 EST 2010" "Mon Dec 06 21:52:56 EST 2010" "" "" ""
"" "" "lobster"
![Page 20: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/20.jpg)
Reviewing assignments
• Approve• Reject• Grant bonus
Word to wise: Preview in Excel!
![Page 21: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/21.jpg)
Worker issues
• Approval rating• Blocked by multiple requestors banned!!!• Too many rejections banned (???)
• My rule: If they made a reasonable attempt to follow the directions, then pay.– If quality is bad, get in touch.
![Page 22: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/22.jpg)
Worker issues
• Indiscriminate rejections…
![Page 23: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/23.jpg)
Qualification Requirements
• Qualification Type ~ HIT Type• Qualification Requirement ~ HIT• Qualification ~ Assignment
• Custom• Built-in• Granted
![Page 24: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/24.jpg)
Informed Consent
![Page 25: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/25.jpg)
Ways to receive results
• Ad-hoc script• Polling• Notification receptor
![Page 26: Crowdsourcing with Amazon Mechanical Turk](https://reader036.fdocuments.net/reader036/viewer/2022062308/56813369550346895d9a7ec6/html5/thumbnails/26.jpg)
Terminology review
• HIT Type• HIT• Assignment (HIT, worker)• Answer part of Assignment
• Qualification Type• Qualification Requirement• Qualification (QR, worker)
HIT Type
HIT