Reputation Systems For Open Collaboration, CACM 2010

Bo Adler, Luca de Alfaro et al.

Nishith Agarwalnishitha@usc.edu

What are Reputation Systems?

• Wikipedia Definition:A reputation system computes reputation scores for a set of objects within a community or domain, based on a collection of opinions other users hold about the objects.

• Why do we need them?•Reputation Systems can help stem abuse of content, and can offer indications of content quality.•In many ways, reputation systems are the on-line equivalent of the body of laws that regulates the real-world interaction of people.

• Who uses reputation systems?

2

Types of Reputation Systems?

3

WikiTrust

• A content-driven reputation system for Wiki authors and content on Wikipedia

• Goals:• Incentivize users to give lasting contributions• Help users and editors increase the quality of content• Offer content consumers a guide to content quality

• Components:• User reputation system: Users gain reputation when they add

content that is preserved by subsequent users.• Content reputation system: Content gains reputation when it

is revised by highly reputed authors.

• Firefox Extension: Lets users view content reputation by changing text background color.

4

WikiTrust – User Reputation System

Contribution quality: Relies on edit distance between revisions.

• -1 if changes made by b are completely reverted• +1 if changes made by b are completely

preserved

5

• Contributions are considered good quality if the change is preserved in subsequent revisions

• User reputation is computed according to quality and quantity of contributions they make

User Reputation: Is proportional to the edit distance and contribution quality of b.

r(B) ≈ d(a,b) + q(b | a,c) + r(C)

• r(B) being the reputation of author B of revision b• r(C) being the reputation of author C of revision c

WikiTrust – Content Reputation System (TextTrust)

• Based on extent to which content was revised, and reputation of users who revised it.

• High content reputation requires consensus from reputed authors.

Basic Algorithm: • Content that is edited is assigned a small fraction of the revision user’s

reputation. • Unedited content gains more reputation.

Some Tweaks:• Ensures that re-arranging or deleting text leaves a low reputation mark• Content reputation cannot exceed the revision user’s reputation• Users cannot raise arbitrary reputation by multiple edits

6

Crowdsensus

• A content-driven reputation system built to analyze user edits to Google Maps

Goals:• To measure accuracy of a user who contributes information• To display accurate details for a business (title, address, phone

etc.)

• Differences from WikiTrust:• There exists a “ground truth”• User reputation is not visible. Hence, no need to keep algorithm

simple.• User identity is stronger

7

Crowdsensus - Algorithm

Structured as fixed point graph algorithm• Vertices are users u, and business attributes a• Edges are attribute values v

• Each user has truthfulness value qu

Algorithm Details:• User vertices send (qu , vu) pairs to value vertices

• An attribute inference algorithm is used to derive probability distribution over values (v1 , v2 ..vn)

• Crowdsensus sends back to user u, the estimate probability that vu is correct

• Different attribute inference algorithms tailored to every attribute type

Comparison to Bayesian Inference Model:• For 1000 attributes, 100 users, 10 attribute values:

– CrowdSensus error rate: 2.8%– Bayesian error rate: 7.9%

• For 1000 attributes, 100 users, 5 attribute values– CrowdSensus error rate: 12.6%– Bayesian error rate: 22%

8

Design Considerations

Uservs.

Content Driven

Visibile to

User?

Weak vs.

Strong Identity

Existence of

Ground Truth

Chronological vs.

Global Updates

Wikipedia

Content Yes Weak No Chronological

Google Maps

Content No Strong Yes Both

Yelp User Yes Strong No Chronological

9

Conclusion

10

Research Directions:•How can reputation systems lead to happy, active, healthy communities?•How can we build reputation systems that meet multiple goals?

Pros

11

Cons

12