beneath under We lost the baby and found her beneath the bed.
Collaborative work beneath the surface Visitors only look at article pages But much of Wikipedia...
-
Upload
julia-king -
Category
Documents
-
view
213 -
download
0
Transcript of Collaborative work beneath the surface Visitors only look at article pages But much of Wikipedia...
Collaborative work beneath the surface
• Visitors only look at article pages• But much of Wikipedia comprised of
other pages– Conflict resolution, coordination, policies and
procedures
Types of work
Direct work Immediately consumable
Indirect workCoordination,
conflict
Maintenance work Reverts, vandalism
Article Talk, user, procedure
Less direct work
• Decrease in proportion of edits to article page
0.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
2001 2002 2003 2004 2005 2006
Edi
t pr
opor
tion
70%
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
2001 2002 2003 2004 2005 2006
Ed
it P
rop
ort
ion
More indirect work
• Increase in proportion of edits to user talk
8%
More indirect work
• Increase in proportion of edits to user talk
• Increase in proportion of edits to procedure
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
2001 2002 2003 2004 2005 2006
Edi
t pr
opor
tion 11
%
More maintenance work
• Increase in proportion of edits that are reverts
00.020.040.060.08
0.10.120.140.160.18
0.2
2001 2002 2003 2004 2005 2006
Ed
it p
rop
ort
ion
7%
More wasted work
• Increase in proportion of edits that are reverts
• Increase in proportion of edits reverting vandalism
00.005
0.010.015
0.02
0.0250.03
2001 2002 2003 2004 2005
Ed
it p
rop
ort
ion
1-2%
Global level
• Coordination costs are growing– Less direct work (articles)+ More indirect work (article talk, user,
procedure)+ More maintenance work (reverts, vandalism)
Kittur, Suh, Pendleton, & Chi, 2007
Article lifespan
• How do articles change over time?• High discussion and coordination
– Kittur et al., 2007; Viegas et al., 2007
• When does this happen?– Hyp 1: Early when articles are growing– Hyp 2: Late when articles are more stable
Article lifespan
User lifespan
• How do users change over time?
Centralization in Wikipedia
• How much centralization?• “Gang of 500” (Jimmy Wales, 2004)
– Small group of ~500 does half the work
• Masses do the work (Aaron Swartz, 2006)
– New users add most of the words
Hypotheses
• Masses dominate• Elite privileged group• Shift from elites to masses
– Technology adoption (Rogers, 1962)
Masses Elites Shift
Elites
• Admins• Editing status (fixed-size)• Editing status (scaling)
Admins
• Waxing and waning of admin influence
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
2001 2002 2003 2004 2005 2006
Pro
port
ion
of to
tal e
dits
mad
e by
adm
ins
Nature News, 2/2007; Kittur, Chi, Pendleton, Suh, Mytkowicz, 2007
Pro
port
ion
of
all
ed
its
Admins
• Similar for changed words
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
2001 2002 2003 2004 2005 2006
Pro
portio
n ch
ange
d w
ords
(adm
ins)
Pro
port
ion
of
word
s ch
an
ged
Elites
• Admins• Editing status (fixed-size)• Editing status (scaling)
Editing status (fixed size)
Elites
• Admins• Editing status (fixed-size)• Editing status (scaling)
Editing status (scaling)
• Proportional influence of elites still high– Though absolute number of elites growing
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
2001 2002 2003 2004 2005 2006
Pro
port
ion
of E
dits
Top 5%
Top 3%
Top 1%
Summary: Centralization
• Centralized elite influence is waning– Decline in admin influence– Decline in data-driven “Gang of 500”
• Decentralized proportional influence remains high– Top 1/3/5% of users account for ~50/70/80%
of edits– The “Bourgeosie”
Challenges for Wikipedia
• Coordination costs• Organization structure• Conflict
Characterizing conflict
Conflict at the article level
• What leads to conflict in articles?• Build a characterization model of article
conflict– Identify page features and metrics
associated with conflict– Automatically identify high-conflict articles
Page metrics
• Chose metrics for identifying conflict in articles– Easily computable, scalable
Metric type Page Type
Revisions (#)Article, talk, article/talk
Page lengthArticle, talk, article/talk
Unique editorsArticle, talk, article/talk
Unique editors / revisions
Article, talk
Links from other articles Article, talk
Links to other articles Article, talk
Anonymous edits (#, %) Article, talk
Administrator edits (#, %)
Article, talk
Minor edits (#, %) Article, talk
Reverts (#, by unique editors)
Article
Defining conflict
• Operational definition for conflict • Revisions tagged controversial
• Conflict revision count
Machine learning
• Predict conflict from page metrics– Training set of “controversial” pages– Support vector machine regression
predicting # controversial revisions (SMOreg; Smola & Scholkopf, 1998)
• Not just conflict/no conflict, but how much conflict
Performance: Cross-validation
• 5x cross-validation, R2 = 0.897
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Predicted controversial revisions
Act
ual c
ontrov
ersial
revi
sion
s
Performance: Cross-validation
• 5x cross-validation, R2 = 0.897
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Predicted controversial revisions
Act
ual c
ontrov
ersial
revi
sion
s
Determinants of conflict
1. —Revisions (talk)2. —Minor edits (talk)3. ˜Unique editors (talk)4. —Revisions (article)5. ˜Unique editors (article)6. —Anonymous edits (talk)7. ˜Anonymous edits (article)
Highly weighted metrics of conflict model:
Identifying untagged articles
• Detect conflicts for unlabeled articles– Majority of articles have never been conflict
tagged
• Testing model generalization– Applied model to untagged articles– Sample of 28 articles rated by 13 expert
Wikipedians
• Significant positive correlation with predicted scores– By rank correlation, p < 0.013 (Spearman’s
rho)
Characterizing conflict
Conflict at the user level
• How can we identify conflict between users?
• Reverts between users as a proxy for user conflict
• Force directed layout to cluster users– Group similar viewpoints– Find conflicts between groups
Dokdo/Takeshima opinion groups
Group A
Group B Group C
Group D
Terry Schiavo
Mediators
Sympathetic to parents
Sympathetic to husband
Anonymous (vandals/spammers)
Cognitive atlas
Visualizing hypotheses
Distributed collaboration
• Lots of people• Each doing a little bit of work• Leads to high quality outcome (i.e., “wisdom
of crowds”)
Francis Galton OxScale
Distributed collaboration
• Applications of distributed collaboration:– Judging: weight of an ox, temperature of a
room– Search: Google PageRank– Predicting: Iowa Electronic Market, Las
Vegas, HP– Filtering: Digg, Reddit– Organizing: del.icio.us
• Common characteristics:– Independent judgments– Independent aggregation
Wikipedia and the wisdom of crowds
• But these are not characteristic of Wikipedia:– Independent judgments– High coordination costs (Kittur et al., 2007)
– Independent aggregation – Competitive aggregation (everyone is editing
the same information)
• To the extent that judgments and aggregation of individual tasks are not independent and instead require coordination and engender conflict, having more editors may not be beneficial and may even be harmful
Travesty of the commoners?
• Increasing size of group generally has negative consequences:– Increased coordination costs– Increased anonymity and social loafing– Decreased attribution and individual reward– More negative social relations– Greater conflict and misbehavior– Loss of control– Cognitive overload
see Bettenhausen, 1991; Levine & Moreland, 1990
Wilkinson & Huberman, 2007
• Examined featured articles vs. non-featured articles– Controlling for PageRank (i.e., popularity)
• Featured articles = more edits, more editors
• “More work, better outcome”: WP similar to other distributed collaboration systems
Nature News (2/27/07)
Problem: Distribution of work
• However, articles can have different distributions of work, even with same edits/editors
• If an article has 1000 edits and 100 editors, it could have:– 1 editor making 901 edits, 99 making 1 edit– 100 editors making 10 edits each
<>
Capturing skew
• Gini coefficient: measures inequality of distribution
• Measure Gini coefficient for each article– Count how many edits each editor makes,
calculate ratio• If an article is driven by few, gini -> 1• If an article is driven by many, gini -> 0
http://en.wikipedia.org/wiki/Gini_coefficient
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 10 100 1000 10000 100000
Edits
Gin
i co
eff
icie
nt
Top15k Page hits
Featured
* Sig difference betw een featured (M=.46) and Top5k (M=.39) gini coeff icients (p < .0001), and betw een Top5k (M=.39) and 5-15k (M=.34, p < .0001)
Old results
0
0.2
0.4
0.6
0.8
1
1 10 100 1000 10000 100000
Edits
Gin
i co
eff
icie
nt
Top15k Page hits
Featured
* Sig difference betw een featured (M=.46) and Top5k (M=.39) gini coeff icients (p < .0001), and betw een Top5k (M=.39) and 5-15k (M=.34, p < .0001)
P(Featured | Gini quintile)
Probability of Being Featured
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5
Gini quintile
P(F
eatu
red
)
ExpertsCrowds
1
10
100
1000
10000
1 10 100 1000 10000 100000
Edits
Un
iqu
e e
dit
ors
Top15k Page hits
Featured
Unique editorsFeat vs. Top15k (M=381), p < .001***
Unique editors x Edits
New results
• Sampled articles at a variety of quality levels– Defined and rated by expert Wikipedians– Hundreds of thousands of articles rated
Cross-sectional analysis
• 900 articles sampled from Start through Featured– Higher quality associated with higher gini,
higher editors
Average of artGini
0
0.1
0.2
0.3
0.4
0.5
0.6
Start-Class B-Class GA-Class A-Class FA-Class
Average of artEditors
050
100150200250300350400
Start-Class
B-Class GA-Class A-Class FA-Class
FA-Class
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 10 100 1000 10000
Number of article editors
Art
icle
Gin
i (0=
equa
l con
trib
s)
FA-Class
A-Class
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 10 100 1000 10000
Number of article editors
Art
icle
Gin
i (0=
equa
l con
trib
s)
A-Class
GA-Class
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 10 100 1000 10000
Number of article editors
Art
icle
Gin
i (0=
equa
l con
trib
s)
GA-Class
B-Class
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 10 100 1000 10000
Number of article editors
Art
icle
Gin
i (0=
equa
l con
trib
s)
B-Class
Start-Class
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 10 100 1000 10000
Number of article editors
Art
icle
Gin
i (0=
equa
l con
trib
s)
Start-Class