Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic...
-
Upload
noah-maloney -
Category
Documents
-
view
219 -
download
4
Transcript of Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic...
![Page 1: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/1.jpg)
Crowdsourcing and All-Pay Auctions
Milan VojnovićMicrosoft Research
Joint work with Dominic DiPalantino
UC Berkeley, July 13, 2009
![Page 2: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/2.jpg)
Examples of Crowdsourcing• Crowdsourcing = soliciting solutions via open calls to
large-scale communities– Coined in a Wired article (’06)
• Taskcn– 530,000 solutions posted for 3,100 tasks
• Innocentive– Over $3 million awarded
• Odesk– Over $43 million brokered
• Amazon’s Mechanical Turk– Over 23,000 tasks
2
![Page 3: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/3.jpg)
Examples of Crowdsourcing (cont’d)
• Yahoo! Answers– Lunched Dec ’05– 60M users / 65M answers (as of Dec ’06)
• Live QnA– Lunched Aug ’06 / closed May ’09– 3M questions / 750M answers
• Wikipedia
3
![Page 4: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/4.jpg)
Incentives for Contribution• Incentives
– Monetary
$$$
– Non-momentary
Social gratification and publicityReputation pointsCertificates and “levels”
• Incentives for both participation and quality
4
![Page 5: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/5.jpg)
Incentives for Contribution (cont’d)• Ex. Taskcn
5
Reward range (RMB)
Cont
est d
urati
onN
umbe
r of s
ubm
issi
ons
Num
ber o
f reg
istr
ants
Num
ber o
f vie
ws
100 RMB $15 (July 09)
![Page 6: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/6.jpg)
Incentives for Contribution (cont’d)• Ex. Yahoo! Answers
6
Points Levels
Source: http://en.wikipedia.org/wiki/Yahoo!_Answers
![Page 7: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/7.jpg)
Questions of Interest
• Understanding of the incentive schemes– How do contributions relate to offered rewards?
• Design of contests– How do we best design contests?– How do we set rewards?– How do we best suggest contests to players and
rewards to contest providers?
7
![Page 8: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/8.jpg)
Strategic User Behavior
• From empirical analysis of Taskcn by Yang et al (ACM EC ’08) – (i) users respond to incentives, (ii) users learn better strategies– Suggests a game-theoretic analysis
8
User Strategies on Taskcn.com User Strategies on Taskcn.com
![Page 9: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/9.jpg)
Outline• Model of Competing Contests
• Equilibrium Analysis– Player-Specific Skills– Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion9
![Page 10: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/10.jpg)
Single Contest Competition
10
c1
c2
c3
c4
R
ci = cost per unit effort or quality produced
contest offeringreward Rplayers
![Page 11: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/11.jpg)
Single Contest Competition (cont’d)
11
Outcome
-c1b1
R - c2b2
-c3b3
-c4b4
c1
c2
c3
c4
b1
b2
b3
b4
R
![Page 12: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/12.jpg)
All-Pay Auction
12
Outcome
-b1
v2 - b2
-b3
-b4
v1
v2
v3
v4
b1
b2
b3
b4
Everyone pays their bid
![Page 13: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/13.jpg)
Competing Contests
13
R1
R2
RJ
...
Rj...
contestsusers
1
2
u
N
),,( ,1, Juuu vvv
juv ,
......
![Page 14: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/14.jpg)
Incomplete Information Assumption
Each user u knows
= total number of usersN
= his own skilluv
= skills are randomly drawn from FF
14
We assume F is an atomless distribution with finite support [0,m]
![Page 15: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/15.jpg)
Assumptions on User Skill1) Player-specific skill
random i.i.d. across u (ex. contests require similar skills or skill determined by player’s opportunity cost)
),,( uu vvv
2) Contest-specific skill
random i.i.d. across u and j (ex. contests require diverse skills)
),,( ,1, Juu vvv
juv ,
uv
15
![Page 16: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/16.jpg)
Bayes-Nash Equilibrium
• Mixed strategy
• EquilibriumSelect contest of highest expected profit
where expectation with respect to “beliefs” about other user skills
)(, vju = prob. of selecting a contest of class j
jub , = bid
16Contest class = set of contests that offer same reward
![Page 17: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/17.jpg)
User Expected Profit
• Expected profit for a contest of class j
v
Ncjjjj dxxFpRvg
0
1)(1)(
= prob. of selecting a contest of class j
jp
= distribution of user skill conditional on having selected contest class j
()jF
17
vn
jn
jjujj dxxFvFvRnvg0
, )()(),(
)),((E)( Mvgvg jj
),1(Bin~ jpNM
![Page 18: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/18.jpg)
Outline• Model of Competing Contests
• Equilibrium Analysis– Player-Specific Skills– Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion18
![Page 19: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/19.jpg)
Equilibrium Contest Selection
m
0
1
2
3
4
5
1v2
v3
v4
2
3
4
skilllevels
contestclasses
19
![Page 20: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/20.jpg)
Threshold Reward
• Only K highest-reward contest classes selected with strictly positive probability
)(
11:max
~],1[
],1[
1
1
RHJ
RiK ii
Ni
1
11
)(
AkkJ
JA
N
A
k RRH
Ak
kA JJ
20kJ = number of contests of class k
![Page 21: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/21.jpg)
Partitioning over Skill Levels
• User of skill v is of skill level l if
KlRH
RJvF
l
lll
N ~,,1 for ,
)(11)(
],1[],1[
11
),[ 1 ll vvv
where
KKlv l ,,~
for ,0
21
![Page 22: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/22.jpg)
Contest Selection
• User of skill l, i.e. with skill selects a contest of class j with probability
Klj
ljR
R
vl
kk
j
j N
N
,,10
,,1)(
1
11
11
),[ 1 ll vvv
22
![Page 23: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/23.jpg)
Participation Rates
• A contest of class j selected with probability
KKj
Kj
R
RH
Jp Nj
K
Kj
,,1~
0
~,,1
)(111
1
1
]~
,1[
]~
,1[
23
• Prior-free – independent of the distribution F
![Page 24: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/24.jpg)
Large-System Limit
• For positive constants
where K is a finite number of contest classes
J
NNlim
kk
N J
J lim
kkN Np lim
Kkkk ,,1 , , ,
KRRR 21
24
![Page 25: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/25.jpg)
Skill Levels for Large System
• User of skill v is of skill level l if
KlR
RvF
l
l
kk
ll
lk
~,,1 for ,log1)( 1
/
],1[
],1[
),[ 1 ll vvv
where
KKlvl ,,1~ for ,0
25
![Page 26: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/26.jpg)
Participation Rates for Large System
• Expected number of participants for a contest of class j
,K,Kj
Kj
R
RK
kk
j
Kj
Kk
1~
0
~,,1log ~
1
/]~
,1[ ]~
,1[
],1[],1[
1
/:max~ iik eRRiKi
kki
26
• Prior-free – independent of the distribution F
![Page 27: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/27.jpg)
Contest Selection in Large System• User of skill l, i.e. with skill selects a
contest of class j with probability
Klj
ljJv lj
,,10
,,11
)( ],1[
),[ 1 ll vvv
m
0
1
2
34
5
123
4
1/3
1/3
1/3
27
• For large systems, what matters is which contests are selected for given skill
![Page 28: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/28.jpg)
Proof Hint for Player-Specific Skills
28
• Key property – equilibrium expected payoffs as showed
vm0 v1v2v3
g1(v)
g2(v)
g3(v)
g4(v)
4321 RRRR
![Page 29: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/29.jpg)
Outline
• Model of Competing Contests
• Equilibrium Analysis– Player-Specific Skills– Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion29
![Page 30: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/30.jpg)
Contest-specific Skills
• Results established only for large-system limit
• Same equilibrium relationship between participation and rewards as for player-specific skills
30
![Page 31: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/31.jpg)
Proof Hints
• Limit expected payoff – For each ],0[ mv
veRvg jjjN
)(lim
• Balancing – Whenever 0j
keReR kjkj all for ,
• Asserted relations for follow from above
),,( 1 K 31
![Page 32: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/32.jpg)
Outline• Model of Competing Contests
• Equilibrium Analysis– Player-Specific Skills– Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion32
![Page 33: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/33.jpg)
System Optimum Rewards
33
K
kkk
K
kkkk RCRU
11
)())((
RR
K
kkk
1
maximise
over
subject to
SYSTEM
• Set the rewards so as to optimize system welfare
![Page 34: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/34.jpg)
Example 1: zero costs(non monetary rewards)
34
Assume are increasing strictly concave functions. Under player-specific skills, system optimum rewards:
()kU
KjN
UcR
N
jj ,,1 ,
)(1
)1(1'
for any c > 0 where is unique solution of
K
kkkU
1
1' )(
• Rewards unique up to a multiplicative constant – only relative setting of rewards matters
![Page 35: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/35.jpg)
Example 1 (cont’d)
35
• For large systems
Assume are increasing strictly concave functions. Under player-specific skills, system optimum rewards:
()kU
KjceR jUj ,,1 ,)(1'
for any c > 0 where is unique solution of
K
kkkU
1
1' )(
![Page 36: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/36.jpg)
Example 2: optimum effort
36
• Consider SYSTEM with
)))(1(1())(( )(Rjjjj
jeRmRR
)))((())(( RVRU jjjjj
)()1()( )(jj
Rj RDeRC j
exerted effort
{cost of
giving Rj (budget constraint)
{
prob. contest attended
{
Utility:
Cost:
![Page 37: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/37.jpg)
Outline• Model of Competing Contests
• Equilibrium Analysis– Player-Specific Skills– Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion37
![Page 38: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/38.jpg)
Taskcn• Analysis of rewards and participation across
tasks as observed on Taskcn– Tasks of diverse categories: graphics, characters,
miscellaneous, super challenge– We considered tasks posted in 2008
38
![Page 39: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/39.jpg)
Taskcn (cont’d)
39
reward
number of views
number of registrants
number of submissions
![Page 40: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/40.jpg)
Submissions vs. Reward
• Diminishing increase of submissions with reward
40
Graphics Characters Miscellaneous
linear regression
![Page 41: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/41.jpg)
Submissions vs. Rewardfor Subcategory Logos
• Conditioning on the more experienced users, the better the prediction by the model
41
any rate once a month every fourth day every second day
• Conditional on the rate at which users submit solutions
model
![Page 42: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/42.jpg)
Same for the Subcategory 2-D
42
any rate once a month every fourth day every second day
model
![Page 43: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/43.jpg)
Conclusion• Crowdsourcing as a system of competing contests
• Equilibrium analysis of competing contests– Explicit relationship between rewards and participations
• Prior-free– Diminishing increase of participation with reward
• Suggested by the model and data
• Framework for design of crowdsourcing / contests
• Base results for strategic modelling– Ex. strategic contest providers
43
![Page 44: Crowdsourcing and All-Pay Auctions Milan Vojnović Microsoft Research Joint work with Dominic DiPalantino UC Berkeley, July 13, 2009.](https://reader035.fdocuments.net/reader035/viewer/2022062511/551475845503462d4e8b6296/html5/thumbnails/44.jpg)
More Information
• Paper: ACM EC ’09
• Version with proofs: MSR-TR-2009-09– http://research.microsoft.com/apps/pubs/default.
aspx?id=79370
44