Adaptive Rewards Mechanism for Sustainable Online Learning Community

Julita Vassileva

Computer Science Department University of Saskatchewan, Canada

Outline

• The problem • Comtella: a link-sharing online community• Encouraging students to rate links• Adaptive rewards mechanism

– Community model and individual model– Adaptive rewards

• Case study• Conclusions

The problem• Why encouraging participation in online communities?

– The majority of systems remain unused– Incentive mechanism is needed

• To encourage participation • However, excessive participation can kill a community

• To ensure sustainability the incentive mechanism needs to:– Discourage excessive participation– Ensure a way to measure the quality of contributions– Encourage timely contributions (when needed by the

community most) and good contributions (from people who tend to have higher standards)

Comtella

• Brief history– 2002: P2P (Gnutella) client for sharing research

papers (as files) in the MADMUC lab (Vassileva @ CoopIS’02, Bretzke & Vassileva @ UM’03)

– 2004: Modified to support a 4th year students in a class on Ethics and IT to share URLs related to each topic/ week) – (Cheng & Vassileva @ ITS2004, @HICSS05)

– 2005: Modified as centralized web-based online community (try it out at http://svaroy.usask.ca:8080/comtella)

back

Motivation Strategies in Comtella• Rewarding participatory acts with points and status

(membership)– Sharing new links, rating links– Points accumulate and result in higher membership for the user

– Visualization of the community showing user participation allowing different views for each kind of activity

• Results - case study in 2004: – Very effective in increasing participation in sharing new papers– But also some low quality papers; excessive participation, high

cognitive loadCheng R., Vassileva, J. (2005) User Motivation and Persuasion Strategy for Peer-to-peer Communities. HICSS'2005 Vassileva, J. (2004) Harnessing P2P Power in the Classroom. Proceedings Intelligent Tutoring Systems, ITS'2004

Summary so far

• Measuring and rewarding participation can stimulate participation

• However, quality tends to deteriorate (some students game the system)

• Therefore, – excessive participation should not be

encouraged – a way for measuring quality and rewarding

high-quality contributions is needed

Encouraging students to rate links

• Modeled after Slashdot– Students with high membership get more

ratings to give out – more power• In addition, each act of rating is rewarded

– By earning c-points (a kind of virtual currency)– By earning participation points (towards

higher membership)

Measuring the Quality of Contributions

• All contributed URLs start at 0 – unlike Slashdot, where it depends on the

“karma” of the user who submitted the post

• The final rating is the sum of all received ratings

• How to judge the quality of a rating?– based on its similarity with other ratings for the

same link

Incentives for rating (1)

• Each act of rating earns c-points for the user• The user can “invest” her c-points to increase

the visibility of the links she contributes

Incentives for rating (2)

• By rating links users also earn participation points towards their membership

• The number of points earned depends on the quality of each rating– Tendency towards the “average” taste?– Is this the taste of the community?

• How to prevent gaming the system?– allow sorting search results by average rating – but do not show the average rating

Adaptive Rewards Mechanism

• Adapting to what?– To the current needs of the community – To the individual quantity and quality

standards of the user• Adapting what?

– The participation points earned by different activities (“economic” incentives)

– The personal message to the user at login

Skip details

Community Model

• Expected / desirable number of links for topic (set by instructor) – Qc– Depends on topic, the time

in the semester, busyness of students etc.

• Community reward factor (depends on time since introducing topic) - Fc

Individual Model (1)• The user reputation in bringing high quality

links: RI = the average summative rating of all links contributed by the user – Users with high RI should be encouraged to

contribute more at any time– Users with low RI should be discouraged, unless

the topic is still fresh and lacks links. • The expected number of links from individual

user for the current topic

I

ICI R

RQQ

Individual model (2)

• Individual reward factor FI

• User reputation in giving high-quality ratings– Computed as 1/EI where

N

RRE

N

iii

I

1

Adaptive rewards mechanism• Varying weights Wi(t) for particular forms of

participation • The user contribution is computed as:

where t – the time of contribution

• The weights depend on the state of the user’s individual model and on the community model at the moment of contribution– Weight for sharing links – Wieght for giving ratings – proportional to the user

reputation for giving high quality ratings 1/EI

n

i

T

ti

i

tWV1 1

)(

ICSS FFWW 0

Adaptive messages to the user

• Communicating the rewards that the user will get for each type of action at the time of login.– user who shares many papers of poor quality

gets a low RI and a small QI , therefore little reward for subsequent contributions.

– the related message would be to contribute less in next period but improve the quality of her contributions.

– Show it

Case study• Comtella used in the “Ethics and IT” class at the UofS

– Jan-April 2005– 32 students

• Two groups of 16 throughout the term• Test - Comtella 1: with adaptive rewards mechanism and c-points• Control - Comtella 2: no adaptive rewards mechanism, no c-points

• Groups formed to have equal gender and Canadian / foreign representation– Test and Control groups are separate communities

• no interaction of shared links, ratings etc. • however, students were in the same classroom for lectures; project teams

across both groups• We compared the numbers of contributions in each group (links,

ratings)• Post-study online questionnaire

Questions and answers (1)• Did the users in the test group (Comtella 1) give more

ratings? – Yes: nearly twice as much as Comtella 2: 1065 vs. 613 ratings (significant)

• Did the summative ratings in Comtella 1 reflect better the quality of the contributed links?– Yes: in Comtella 1, 56% (9 users) felt that the final summative ratings that

their links received reflect fairly their quality, while in Comtella 2, only 25% (4 users) thought so.

• Did the users in Comtella 1 tend to share links earlier in the week?– Yes: users in Comtella 1 shared 71.3% of their contributions in the first 3

days after introducing the topic; users in Comtella 2 shared 60.6% of their contributions in the first 3 days. The difference was significant for all topics and ranged between 7-14%.

• Did the users in Comtella 1 share the number of links that was expected from them? – To some extent: about half of the users did.

Questions and answers (2)• Did the users in Comtella 1 participate more actively in general?

– Yes: they read more papers (3419 vs. 2416) and logged in the system more frequently (1714 vs. 982).

• Is there a significant difference with respect to the total number of contributed links between the test and the control group? – No: 613 in Comtella 1 versus 587 in Comtella 2– There was no excessive paper contribution in either case.

• Did the users in Comtella 1 contribute links with higher quality?• Is there a correlation between the ratings of links and the times they

were chosen for summarizing?– Yes: strong correlation: – 0.861 for links with rating > 0,

0.928 for links with rating >1, 0.983 for links with rating > 2, 1 for links with rating > 3

Discussion• Incorporating an incentive mechanism can stimulate a

desired behaviour in an online community – in our case, the c-points stimulated ratings– can be useful for other systems depending on user ratings, e.g.

collaborative filtering systems (Movie Lens, Amazon)• An adaptive rewards mechanism can orchestrate a desired

pattern of collective behaviour– in our case, the time-adaptation of the rewards stimulated users to

make contributions earlier• It is important to make the user aware of the rewards for

different actions at any given time• More experiments are needed to show effect of adaptation

to individual pattern and impact on the quality / quality perception

Future work

• Deal with the subjectivity of ratings• Adapt the mechanism to motivate

contributions in a different kind of community, – a research group or a group of graduate

students sharing research papers, – a community of researchers (the AIED and /or

the UM community?) – a community for girls and women in science

and engineering

More information

• Try Comtella at: – http://svaroy.usask.ca:8080/comtella– See how it is used for sharing research papers in the

MADMUC lab: http://svaroy.usask.ca:8080/aied

• See other papers, follow up project:– http://bistrica.usask.ca/madmuc/peer-motivation.htm– Or just Google for “Comtella”

Back