Post on 06-Aug-2015
Expertise Profiling
03 / 20
Gamification
04 / 20
๏ question-answering quiz๏ IR publications
(1111 papers from SIGIR, WWW, CIKM, KDD & WSDM)
๏ two difficulty modes ๏ max 3 mistakes; time limit๏ goal: answer as many questions as possible๏ motivation for players: position on the leader board
Research questions
05 / 20
๏ Which level of difficulty is preferred?
๏ Does a competitive element, such as a leader board, increase the level of engagement?
๏ When do users stop playing?
๏ Do users return to play again? After how long?
๏ What types of players can we identify?
๏ Are more cited papers also more easily recognized?
๏ Are more popular authors also more easily recognized?
๏ Do people prefer to play anonymously?
Usage stats
09 / 20
Duration of measurements:
# webpage visitors:
# game players:
# games played:
# games (beginner / advanced):
# avg. #games per player:
5 days (Jan 31 - Feb 4, 2015)
302 (from 33 countries)
116
387
347 / 39
3.34
Analysis of resultsby answers
10 / 20
Time limit (both game modes):
Avg. time to answer (beginner / advanced):
15 s
6.7 s / 8.5 s
Analysis of resultsby answers
11 / 20
Score distribution
a) beginner mode b) advanced mode
Analysis of resultsby players
12 / 20
Returning players
# players who played more than 1x:
# games played within 1 hour:
56
42
Time elapsed between games
Analysis of resultsby players
13 / 20
Player type: Jumper
Analysis of resultsby players
14 / 20
Player type: Give-uper
Analysis of resultsby players
15 / 20
Player type: Fighter
Analysis of resultsby players
16 / 20
Player type: Achiever
Analysis of resultsby papers
17 / 20
Paper’s recognition ratioNumber of times the publication was successfully recognized by playersdivided by number of times it was shown to players.
Citation counts vs. recognition ratio
Analysis of resultsby authors
18 / 20
Author’s recognition ratioNumber of times the author’s publications was successfully recognized by players divided by total number of times her publications were shown to players.
Number of author’s publications vs. recognition ratio.
Observations
19 / 20
๏ Learning
The game was useful for learning about new publications.
๏ Unfair behavior
The best scoring user had the longest response time.
๏ Head-start
A user was restarting the game until he was able to answer the first question correctly.
๏ Engaging users
It is important to keep the user stay in the game as long as possible when she comes for the first time.
๏ Identity
Some people (∼ 10%) opted to use their full civil name as opposed to a nickname.
Conclusions
20 / 20
๏ Could we make an assessment exercise, in the context of expertise profiling, more appealing for users?
Yes.
๏ We gathered valuable data about authors and publications.
Future work ๏ Introduce a controlled bias into selection of alternative answers.
๏ Adjust the difficulty of the questions as the game progresses.
๏ New game modes.
๏ Expand to other research fields.
Questions?
Thank you
Credits
© Luboš Volkov
© iconsmind.com
© Cédric Villain
© convoy
© baabullah hasan
© Luis Prado
© Mark Shorter
© Gerhard Meier
Presented by Jenda Rybák at ECIR 2015, Wien