Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis...
-
date post
22-Dec-2015 -
Category
Documents
-
view
218 -
download
2
Transcript of Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis...
![Page 1: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/1.jpg)
Approval-rating systems that never
reward insincerity
Rob LeGrandWashington University in St. Louis
(now at Bridgewater College)
Ron K. CytronWashington University in St. Louis
COMSOC ’083 September 2008
![Page 2: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/2.jpg)
2
Approval ratings
![Page 3: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/3.jpg)
3
Approval ratings
• Aggregating film reviewers’ ratings– Rotten Tomatoes: approve (100%) or disapprove (0%) – Metacritic.com: ratings between 0 and 100– Both report average for each film– Reviewers rate independently
![Page 4: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/4.jpg)
4
Approval ratings
• Online communities– Amazon: users rate products and product reviews– eBay: buyers and sellers rate each other– Hotornot.com: users rate other users’ photos– Users can see other ratings when rating
• Can these “voters” benefit from rating insincerely?
![Page 5: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/5.jpg)
5
Approval ratings
![Page 6: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/6.jpg)
6
Average of ratings
9.0,8.0,8.0,7.0,4.0v 9.0,8.0,8.0,7.0,4.0r
0 172.0
outcome: 72.0)( vfavg
data from Metacritic.com: Videodrome (1983)
![Page 7: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/7.jpg)
7
Average of ratings
9.0,8.0,8.0,7.0,0v 9.0,8.0,8.0,7.0,4.0r
0 164.0
outcome: 64.0)( vfavg
Videodrome (1983)
![Page 8: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/8.jpg)
8
Another approach: Median
9.0,8.0,8.0,7.0,4.0v 9.0,8.0,8.0,7.0,4.0r
0 18.0
outcome: 8.0)( vfmed
Videodrome (1983)
![Page 9: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/9.jpg)
9
Another approach: Median
9.0,8.0,8.0,7.0,0v 9.0,8.0,8.0,7.0,4.0r
0 18.0
outcome: 8.0)( vfmed
Videodrome (1983)
![Page 10: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/10.jpg)
10
Another approach: Median
• Immune to insincerity– voter i cannot obtain a better result by voting– if , increasing will not change– if , decreasing will not change
• Allows tyranny by a majority– – – no concession to the 0-voters
ii rv imed vvf )(
imed vvf )( iv
iv
1,1,1,1,0,0,0v1)( vfmed
)(vfmed
)(vfmed
![Page 11: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/11.jpg)
11
Declared-Strategy Voting[Cranor & Cytron ’96]
electionstate
cardinal
preferences
rational
strategizer
ballot
outcome
![Page 12: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/12.jpg)
12
Declared-Strategy Voting[Cranor & Cytron ’96]
electionstate
cardinal
preferences
rational
strategizer
ballot
outcome
• Separates how voters feel from how they vote• Levels playing field for voters of all sophistications• Aim: a voter needs only to give sincere preferences
sincerity strategy
![Page 13: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/13.jpg)
13
Average with Declared-Strategy Voting?
• Try using Average protocol in DSV context
• But what’s the rational Average strategy?• And will an equilibrium always be found?
electionstate
cardinal
preferences
rational
strategizer
ballot
outcome
![Page 14: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/14.jpg)
14
Rational [m,M]-Average strategy
• Allow votes between and• For , voter i should choose to move
outcome as close to as possible• Choosing would give• Optimal vote is
• After voter i uses this strategy, one of these is true:– and– – and
0m
)),,min(max( Mmvnrvij jii
iavg rvf )(
ij jii vnrv
iavg rvf )(
Mvi
mvi iavg rvf )(
iavg rvf )(
1Mni 1 iv
ir
![Page 15: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/15.jpg)
15
Equilibrium-finding algorithm
0 1
9.0,8.0,8.0,7.0,4.0r
72.0
9.0,8.0,8.0,7.0,4.0v
Videodrome (1983)
![Page 16: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/16.jpg)
16
Equilibrium-finding algorithm
01
9.0,8.0,8.0,7.0,4.0r
0,0,0,0,0v
![Page 17: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/17.jpg)
17
Equilibrium-finding algorithm
0 1
9.0,8.0,8.0,7.0,4.0r
2.0
1,0,0,0,0v
![Page 18: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/18.jpg)
18
Equilibrium-finding algorithm
0 1
9.0,8.0,8.0,7.0,4.0r
4.0
1,1,0,0,0v
![Page 19: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/19.jpg)
19
Equilibrium-finding algorithm
0 1
9.0,8.0,8.0,7.0,4.0r
6.0
1,1,1,0,0v
![Page 20: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/20.jpg)
20
• Is this algorithm guaranteed to find an equilibrium?
Equilibrium-finding algorithm
0 1
9.0,8.0,8.0,7.0,4.0r
7.0
1,1,1,5.0,0vequilibrium!
![Page 21: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/21.jpg)
21
• Is this algorithm guaranteed to find an equilibrium?• Yes!
Equilibrium-finding algorithm
0 1
9.0,8.0,8.0,7.0,4.0r
7.0
1,1,1,5.0,0vequilibrium!
![Page 22: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/22.jpg)
22
• These results generalize to any range
Expanding range of allowed votes
1 2
9.0,8.0,8.0,7.0,4.0r
8.0
2,2,2,1,1 v
![Page 23: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/23.jpg)
23
• Will multiple equilibria always have the same average?
Multiple equilibria can exist
outcome in each case:
7.0)( vfavg
1,1,1,5.0,0v 9.0,8.0,7.0,7.0,4.0r
1,1,9.0,6.0,0v
1,1,75.0,75.0,0v
![Page 24: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/24.jpg)
24
• Will multiple equilibria always have the same average?• Yes!
Multiple equilibria can exist
outcome in each case:
7.0)( vfavg
1,1,1,5.0,0v 9.0,8.0,7.0,7.0,4.0r
1,1,9.0,6.0,0v
1,1,75.0,75.0,0v
![Page 25: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/25.jpg)
25
Average-Approval-Rating DSV
9.0,8.0,8.0,7.0,4.0v 9.0,8.0,8.0,7.0,4.0r
0 17.0
outcome: 7.0)1,0,( vfaveq
Videodrome (1983)
![Page 26: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/26.jpg)
26
• AAR DSV is immune to insincerity in general
Average-Approval-Rating DSV
9.0,8.0,8.0,7.0,0v 9.0,8.0,8.0,7.0,4.0r
0 1
outcome: 7.0)1,0,( vfaveq
7.0
![Page 27: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/27.jpg)
27
• Expanded vote range gives wide range of AAR DSV systems:
• If we could assume sincerity, we’d use Average• Find AAR DSV system that comes closest• Real film-rating data from Metacritic.com
– mined Thursday 3 April 2008– 4581 films with 3 to 44 reviewers per film– measure root mean squared error
• Perhaps we can come much closer to Average than Median or [0,1]-AAR DSV does
Evaluating AAR DSV systems
10 a 10 b)(, vba
![Page 28: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/28.jpg)
28
Evaluating AAR DSV systems
5.0,aRMSE
a
3240.0aminimum at
5.0b
![Page 29: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/29.jpg)
29
Evaluating AAR DSV systems: hill-climbing
a
3647.0aminimum at
4820.0b
4820.0,aRMSE
![Page 30: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/30.jpg)
30
Evaluating AAR DSV systems: hill-climbing
4820.0bminimum at
3647.0a
bRMSE ,3647.0
b
![Page 31: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/31.jpg)
31
Evaluating AAR DSV systems
)(4820.0,3647.0 v
)(vfavg
![Page 32: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/32.jpg)
32
AAR DSV: Future work
• New website: trueratings.com– Users can rate movies, books, each other, etc.– They can see current ratings without being tempted to
rate insincerely– They can see their current strategic proxy vote
• Richer outcome spaces– Hypercube: like rating several films at once– Simplex: dividing a limited resource among several uses– How assumptions about preferences are generalized is
important
Thanks! Questions?
![Page 33: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/33.jpg)
33
What happens at equilibrium?
• The optimal strategy recommends that no voter change
• So• And
– equivalently,
• Therefore any average at equilibrium must satisfy two equations:– (A)– (B)
1)( ii vrvi
ii rvvi 0)(0)( ii vrvi
irvinv : nvrvi i :
![Page 34: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/34.jpg)
34
Proof: Only one equilibrium average
irinA :)( nriB i :)(
212211 )()()()( BABA
• Theorem:
• Proof considers two symmetric cases:– assume– assume
• Each leads to a contradiction
21 12
![Page 35: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/35.jpg)
35
Proof: Only one equilibrium average
21 case 1:
ii rri 12)( ii riri 12 :: ii riri 12 ::
irin 22 : nri i 11:
nririn ii 1122 :: nn 12
12 21 , contradicting
)( 2A)( 1B
![Page 36: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/36.jpg)
36
Proof: Only one equilibrium average
21 Case 1 shows that
Case 2 is symmetrical and shows that 12
21 Therefore
Therefore, given , the average at equilibrium is uniquer
![Page 37: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/37.jpg)
37
An equilibrium always exists?
• At equilibrium, must satisfy
Given a vector , at least one equilibrium indeed always exists.
A particular algorithm will always find an equilibrium for any . . .
)),,min(max()( Mmvnrviij jii
v
r
r
![Page 38: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/38.jpg)
38
An equilibrium always exists!
Equilibrium-finding algorithm:• sort so that• for i = 1 up to n do
• Since an equilibrium always exists, average at equilibrium is a function, .
• Applying to instead of gives a new system, Average-Approval-Rating DSV.
r
)),,)(min(max( Mmminvnrvik kii
ji rrji )(
),,( Mmrfaveq
v
r
aveqf
(full proof and more efficient algorithm in dissertation)
![Page 39: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/39.jpg)
39
• What if, under AAR DSV, voter i could gain an outcome closer to ideal by voting insincerely ( )?
• It turns out that Average-Approval-Rating DSV is immune to strategy by insincere voters.
• Intuitively, if , increasing will not change .
ii rv
Average-Approval-Rating DSV
iaveq vMmvf ),,(
iv),,( Mmvfaveq
![Page 40: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/40.jpg)
40
• If ,– increasing will not change .– decreasing will not increase .
• If ,– increasing will not decrease .– decreasing will not change .
• So voting sincerely ( ) is guaranteed to optimize the outcome from voter i’s point of view
AAR DSV is immune to strategy
iiaveq rvMmvf ),,(
),,( Mmvfaveq
),,( Mmvfaveq
iviv
iiaveq rvMmvf ),,(
iv
iv
),,( Mmvfaveq
),,( Mmvfaveq
ii rv
(complete proof in dissertation)
![Page 41: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/41.jpg)
41
• [m,M]-AAR DSV can be parameterized nicely using a and b, where and :
mMa
1
mM
mb
1
a
bbM
1
a
bbm
Parameterizing AAR DSV
x
bb
x
bbvfv aveq
axba
1,,lim)(,
10 a 10 b
![Page 42: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/42.jpg)
42
• For example:
Parameterizing AAR DSV
)1,0,()(,1 vfv aveqb
vfv med
)(
2
1,0
11,10,)(2
1,
21
1 vfv aveq
vv
min)(1,0
vv
max)(0,0
2,1,)(2
1,3
1 vfv aveq
![Page 43: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/43.jpg)
43
• Real film-rating data from Metacritic.com– mined Thursday 3 April 2008– 4581 films with 3 to 44 reviewers per film
Evaluating AAR DSV systems
10 a 10 b
2,, vfvvSE avgbaba
V
VV
v
vba
ba v
vSEvRMSE
,
,
![Page 44: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/44.jpg)
44
Higher-dimensional outcome space
• What if votes and outcomes exist in dimensions?
• Example:• If dimensions are independent, Average, Median
and Average-approval-rating DSV can operate independently on each dimension– Results from one dimension transfer
1d
1010:, 2 yxyx
![Page 45: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu.](https://reader036.fdocuments.net/reader036/viewer/2022062715/56649d7e5503460f94a6153c/html5/thumbnails/45.jpg)
45
Higher-dimensional outcome space
• But what if the dimensions are not independent?– say, outcome space is a disk in the plane:
• A generalization of Median: the Fermat-Weber point [Weber ’29]
– minimizes sum of Euclidean distances between outcome point and voted points
– F-W point is computationally infeasible to calculate exactly [Bajaj ’88] (but approximation is easy [Vardi ’01])
– cannot be manipulated by moving a voted point directly away from the F-W point [Small ’90]
1:, 222 yxyx