CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 ·...
51
CS 5306 INFO 5306: Crowdsourcing and Human Computation Lecture 4 8/31/17 Haym Hirsh
Transcript of CS 5306 INFO 5306: Crowdsourcing and Human Computationhirsh/5306/Lecture4.pdf · 2017-09-05 ·...
CS 5306INFO 5306:
Crowdsourcing andHuman Computation
Lecture 48/31/17
Haym Hirsh
Upcoming Speakers
• Thursday, Aug 31, 4:15 (after class)Louis Hyman. The Return of The Independent Workforce: The History and The Future of Work(extra credit)
• Thursday, Sep 7, 4:15 (after class)Henry Kautz, Mining Social Media to Improve Public Health(extra credit)
• Tuesday, Sep 12 (in class)Serge Belongie
• Thursday, Sep 28, 4:15 (after class)Michael Bernstein(extra credit)
“Prehistory” of Human Computation
Characterizing Crowdsourcing and Human Computation Systems
• “Overt” vs “Covert”: Are human participants explicitly participating to achieve the collective outcomes, or is some form of mining of human activity achieving the collective outcomes
– Overt: Amazon reviews, Wikipedia
– Covert (“Crowd Mining”): Google, Amazon recommendations
Characterizing Crowdsourcing and Human Computation Systems
• What are they doing?– Collecting
– Collaborative Creation
– Smartest in the Crowd
– Collaborative Decisions
– Micro-Labor
– Mining User Behavior• Search logs
• Social media
Characterizing Crowdsourcing and Human Computation Systems
• What are they doing?– Collecting
– Collaborative Creation
– Smartest in the Crowd
– Collaborative Decisions
– Micro-Labor
– Mining User Behavior• Search logs
• Social media
Characterizing Crowdsourcing and Human Computation Systems
• What are they doing?– Collecting
– Collaborative Creation
– Smartest in the Crowd
– Collaborative Decisions
– Micro-Labor
– Mining User Behavior• Search logs
• Social media
Characterizing Crowdsourcing and Human Computation Systems
• What are they doing?– Collecting
– Collaborative Creation
– Smartest in the Crowd
– Collaborative Decisions
– Micro-Labor
– Mining User Behavior• Search logs
• Social media
Characterizing Crowdsourcing and Human Computation Systems
• What are they doing?– Collecting
– Collaborative Creation
– Smartest in the Crowd
– Collaborative Decisions
– Micro-Labor
– Mining User Behavior• Search logs
• Social media
“Prediction Markets”
Characterizing Crowdsourcing and Human Computation Systems
• What are they doing?– Collecting
– Collaborative Creation
– Smartest in the Crowd
– Collaborative Decisions
– Micro-Labor
– Mining User Behavior• Search logs
• Social media
Copyright 2011 Haym Hirsh
“Games with a Purpose”
• Introduction: Broad overview of collective intelligence, a framework for understanding it, and its various connections to machine learning– Animals to humans images/videos
– CI mosaic (icons, then list of keywords)
– CI, crowdsourcing, human computation
– Overt vs covert• Collaborative creation
• Collaborative decision making
• Smartest in the crowd / contests
• HC and micro-labor markets
• Crowd mining
– Roles of Machine Learning
Characterizing Crowdsourcing and Human Computation Systems
• What are they doing?– Collecting
– Collaborative Creation
– Smartest in the Crowd
– Collaborative Decisions
– Micro-Labor
– Mining User Behavior• Search logs
• Social media
5 September 2017 Copyright 2011 Haym Hirsh 41
5 September 2017 Copyright 2011 Haym Hirsh 42
5 September 2017 Copyright 2011 Haym Hirsh 43
5 September 2017 Copyright 2011 Haym Hirsh 44
5 September 2017 Copyright 2011 Haym Hirsh 45
“Games with a Purpose”Luis von Ahn
IEEE Computer, June 2006
“A game-theoretic analysis of games with a purpose”S. Jain and D. Parkes
Internet and Network Economics, pp.342-350, 2008
Important Game Theory Concepts
• Nash Equilibrium: A strategy for each player to take actions wherein if either player changes his/her strategy, the outcome will be worse for that actor
• Bayes Game: Each player has information that is unknown to the other player, but the other player has a probability distribution over what that information might be
• Bayes Nash Equilibrium: If either player changes his/her strategy, the expected value of the outcome will be worse, given the probability distribution
Result
• The ESPGame incentivizes players to take a strategy of playing the most common words first• It’s a Bayes Nash Equilibrium
• It yields the best response even for playing otherwise
• This might not match up with the goals of the game designer
Readings for Next Time
• Yu, L., André, P., Kittur, A. and Kraut, R., 2014, February. A comparison of social, learning, and financial strategies on crowd engagement and output quality. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing (pp. 967-978). ACM.
• Mason, Winter, and Duncan J. Watts. "Financial incentives and the performance of crowds." Proceedings HComp 2009. http://crowdsourcing-class.org/readings/downloads/econ/financial-incentives-and-the-performance-of-crowds.pdf
