Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.
-
Upload
ashley-greene -
Category
Documents
-
view
216 -
download
2
Transcript of Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.
![Page 1: Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070400/56649f155503460f94c2ae73/html5/thumbnails/1.jpg)
Crowdsourcing
Nathan McFarlandIsaac Nichols
February 26th, 2013
![Page 2: Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070400/56649f155503460f94c2ae73/html5/thumbnails/2.jpg)
Agenda
• Crowdsourcing History and Background
• Crowdsourcing Demo
• Worker Quality Control Theories and Tactics
• Crowdsourcing War Stories
![Page 3: Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070400/56649f155503460f94c2ae73/html5/thumbnails/3.jpg)
Isaac’s BackgroundFormer UW/CSE Student. Worked for Microsoft & then Amazon after graduation.
Responsible for launching Amazon’s Mechanical Turk on Nov 2005
Sold crowdsourcing consulting business, TurkForce, to CrowdFlower in 2010
Founded Chirpsy in 2011…
![Page 4: Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070400/56649f155503460f94c2ae73/html5/thumbnails/4.jpg)
Nathan’s Background• Started Career by working for 7 startups in
6 years, doing all manner code.
• Followed that with a stint at Los Almos National Labs - doing a bunch of more academic things.
• Founded CastingWords in Nov 2005 right when MTurk launched, ~7 year ago.
![Page 5: Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070400/56649f155503460f94c2ae73/html5/thumbnails/5.jpg)
What is crowdsourcing?
Crowdsourcing is …“Engaging distributed groups of people with an open call to complete a task.”
— Jeff Howe 2006
![Page 6: Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070400/56649f155503460f94c2ae73/html5/thumbnails/6.jpg)
What is crowdsourcing?
Crowdsourcing is …“Engaging distributed groups of people with an open call to complete a task.”
— Jeff Howe 2006
![Page 7: Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070400/56649f155503460f94c2ae73/html5/thumbnails/7.jpg)
What is crowdsourcing?
Crowdsourcing is …“Engaging distributed groups of people with an open call to complete a task.”
— Jeff Howe 2006
![Page 8: Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070400/56649f155503460f94c2ae73/html5/thumbnails/8.jpg)
What is crowdsourcing?
Crowdsourcing is …“Engaging distributed groups of people with an open call to complete a task.”
— Jeff Howe 2006
![Page 9: Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070400/56649f155503460f94c2ae73/html5/thumbnails/9.jpg)
![Page 10: Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070400/56649f155503460f94c2ae73/html5/thumbnails/10.jpg)
Crowd Labor
• Definition: Leveraging of a distributed virtual labor pool, available on-demand to fulfill a range of tasks from simple to complex
• Simple Micro-task Crowdsourcing
• Complex Task Crowdsourcing
![Page 11: Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070400/56649f155503460f94c2ae73/html5/thumbnails/11.jpg)
History Lesson:The Original Mechanical Turk
![Page 12: Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070400/56649f155503460f94c2ae73/html5/thumbnails/12.jpg)
5 Minute Mechanical Turk Demo…
![Page 13: Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070400/56649f155503460f94c2ae73/html5/thumbnails/13.jpg)
Worker Quality Control
• Plurality / Majority Rules• Gold Standard Questions / Known Answers• Peer Review / Grader Validation Workflows• Algorithmic Approaches (Machine Learning)• Worker Reputation• Submission Credibility (Dynamic Quality Control)
![Page 14: Crowdsourcing Nathan McFarland Isaac Nichols February 26 th, 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070400/56649f155503460f94c2ae73/html5/thumbnails/14.jpg)
Crowdsourcing War Stories
• The Good…– Workers can be extremely helpful and proactive
• The Bad...– Mistakes can have long term consequences
• The Ugly… – Unpredictable Fraud