KSE801: Human Computation and Crowdsourcing

Uichin LeeSept. 5, 2011

Knowledge Service Engineering Department

The Rise of CrowdsourcingBy Jeff Howe (Wired Magazine, 2006)

Remember outsourcing? Sending jobs to India and China is so 2003.

The new pool of cheap labor: everyday people using their spare

cycles to create content, solve problems, even do corporate R&D.

The Rise of Crowdsourcing

• The Professional• The Packager• The Tinkerer• The Masses• And the Age of the Crowd

The Professional

• A story of “Claudia Menashe”– A project director at the National

Health Museum in Washington, DC– Putting a series of interactive kiosks

devoted to potential pandemics like the avian flu

– An exhibition designer created a plan for the kiosk; now she wants to have images to accompany the text..• Hire a photographer?• Pre-existing images—stock photograph

The Professional

• She ran across a stock photo collection by Mark

I’ll give you some discount: how about

$100-$150 per photograph?

That’s about half of what a cooperate client would

pay!

Great! I’ll buy 4 images!!

We don’t have much money…

Claudia

Mark

The Professional

One dollar!! That’s a steal!!

iStockphoto: a marketplace for the work of amateur photographers (e.g., homemakers, students, engineers, dancers); over 20,000 contributors which charge about $1 to $5 per basic image

The Packager

• Viral videos; how to repurpose content to make compelling TV on a budget?

• Web Junk 20 at VH1: American television program in which VH1 and iFilm collaborate to highlight the twenty funniest and most interesting clips collected from the Internet that week

• Michael Hirschorn (creator of Web Junk 20)– “I knew we offered something YouTube couldn’t; television.

Everyone wants to be on TV”• TV 2.0 – user generated content

– As user generated TV matures, the users will become more proficient and the networks better at ferreting out the best of the best..

The Tinkerer: The Future of Cooperate R&D

• InnoCentives – Launched in 2001 to connect with brainpower outside the company – Companies pay solvers anywhere from $10,000 to $100,000 per solution– Jill Panetta (CSO) says

• More than 30% of the problems are solved! • The odds of a solver’s success increased in the fields in which they had no formal expertise• “The strength of weak ties”– Mark Granovetter

• Similar Services:

• Ed Melcarek– On most Saturdays, Melcarek attacks problems that have stumped some of the best

cooperate scientists at Fortune 100 companies– “not bad for a few weeks’ work” (e.g., Colgate problem: $25,000)

• P&G’s R&D: – “We have 9,000 people on our R&D staff and up to 1.5 million researchers working

through our external networks”

The Masses• The Turk:

– The first machine capable of beating a human at chess, built around the late 1760s by a Hungarian nobleman named Wolfgan von Kempelen

• Amazon’s Mechanical Turk – Crowdsourcing for the masse (no specific

talents required)– Web based marketplace that helps companies

to find people to perform “human intelligence tasks” (HITs) computers are lousy at

– Examples: identifying items in a photo, skimming real estate documents to find identifying information, writing short product description

– HITs cost from a few cents to a few dollars or more

“Human Intelligence inside”

Our focus in this class: The Masses – Labor Marketplaces, Games, Ubiquitous Sensing, Social Networking/Q&A

The Age of the Crowd• Distributed computing projects: UC Berkeley’s SETI@home?

– Tapping into the unused processing power of millions of individual computers

• “Distributed labor networks”– Using the Internet (and Web 2.0) to exploit the spare processing power of

millions of human brains

• Successful examples?– Open source software: a network of passionate, geeky volunteers could write

code just as well as highly paid developers at Microsoft or Sun Microsystems– Wikipedia: creating a sprawling and surprisingly comprehensive online

encyclopedia– eBay, Facebook: can’t exist without the contributions of users

The Age of the Crowd

• The productive potential of millions of plugged-in enthusiasts is attracting the attention of old-line business too

• For the last decade or so, companies have been looking overseas for cheap labor

• But now it doesn’t matter where the laborers are, as long as they are connected to the Internet

The Age of the Crowd

• Technological advances in everything (from product design software to digital video cameras) are breaking down the cost barriers that once separated amateurs from professionals

• Crowds (e.g., hobbyists, part-timers, dabblers) now suddenly have a market for their efforts

• Smart companies in industries tap the latent talent of the crowd

“The labor isn’t always free, but it costs a lot less than paying traditional employees. It’s not outsourcing: it’s crowdsourcing”

Course Information

Lecture Schedule• W1: Introduction (taxonomy and survey)• W2: Mechanized labor market places (Amazon’s M-Turk, Taskcn)• W3: Using M-Turk for research experiments• W4-5: M-Turk apps: HCI, DB, search• W6: Collaboration, creativity, ethics of crowdsourcing• W7: Task/workflow design• W7: Work quality in the labor marketplaces• W9: Pricing and incentives in the labor marketplaces• W10: Task search and recommendation• W10: Human computation based games (e.g., reCAPTCHA)• W11: Ubiquitous crowdsourcing (smartphones, social networks)• W12: Task planning and incentives in ubiquitous crowdsourcing• W13: Social Q&A systems (Naver KIN, Yahoo! Answer)• W14: Quality and incentives social Q&A systems• W14: Q&A over social networks (Aardvark, Quora)• W15: Real-time Q&A systems

LaborMarketplace

Ubiquitouscrowdsourcing

Games

SocialQ&A

Systems

Logistics

• Grading policy– Mid-term exam: 30%– Quiz: 10% – Critiques: 10%– Course project: 40%– Class participation: 10% (volunteer presentation)

• Office hours:– M/W: 5:20-6PM– By appointments

Critique Guidelines• Provide a summary of the article in your own words (few

sentences) • Provide a brief description of the results based on your

perspectives; please do not simply repeat the words in the paper, but try to explain things on your words in a less formal way

• State the main concept and key contributions (how the work differs from existing work)

• State the strength of the paper (how well the paper is formulated/written? what parts do you like? how the paper improves the state-of-the-arts?)

• State the weakness of the paper (how this paper can be improved? any follow-up questions to pursue?)

Course Project Guidelines

• A team of two students (or alone)• Any topic related to human computation and crowdsourcing is

OK– Project ideas/areas will be also posted at the end of this week

• Detailed proposal by Sept 23, 2011• Mid-term presentation (progress report)• Final presentation (final’s week)• Final report submission (IEEE double column, <= 8 pages) by

Dec 23, 2011– Extra 10 points if the paper is submitted to 1st Workshop on “Social

Network Mining and Human Computation” (held at KAIST in January 2012)

Learning Objectives• Knowledge

– Factual knowledge– Theories and principles– Professional skills and viewpoints– Discipline’s methods

• Capabilities– Thinking and problem solving– Creative capacities– Effective communications

• Personal development– Self-reliance, self-discipline– Interests, talents, values, etc.– General liberal education

Next Class (Wed, Sept. 7, 2011)

• Introduction to human computation and crowdsourcing (taxonomy and survey)

• Reading assignment: – “Human Computation: A Survey and Taxonomy of

a Growing Field,” Alexander J. Quinn, Benjamin B. Bederson, CHI 2011