Post on 18-Nov-2014
description
8/27/2011 www.insemtives.eu 1
Combining human and computational intelligence for
collaborative knowledge creationElena Simperl
Talk at the IEEE International Conference on Intelligent Computer Communication and Processing, Cluj-Napoca, Romania
Insemtives in a nutshell• Many aspects of semantic content authoring naturally rely on human
contribution.
• Motivating users to contribute is essential for semantic technologies to reach critical mass and ensure sustainable growth.
• Insemtives works on – Best practices and guidelines for incentives-compatible technology design.– Enabling technology to realize incentivized semantic applications.– Showcased in three case studies: enterprise knowledge management;
services marketplace; multimedia management within virtual worlds.
www.insemtives.eu 2
Incentives and motivators
• Motivation is the driving force that makes humans achieve their goals.
• Incentives are ‘rewards’ assigned by an external ‘judge’ to a performer for undertaking a specific task.– Common belief (among
economists): incentives can be translated into a sum of money for all practical purposes.
• Incentives can be related to both extrinsic and intrinsic motivations.
• Extrinsic motivation if task is considered boring, dangerous, useless, socially undesirable, dislikable by the performer.
• Intrinsic motivation is driven by an interest or enjoyment in the task itself.
What is different about semantic systems?
• Semantic Web toolsvs applications. – Intelligent (specialized)
Web sites (portals) with improved (local) search based on vocabularies and ontologies.
– X2X integration (often combined with Web services).
– Knowledge representation, communication and exchange.
What do you want your users to do?
• Semantic applications– Context of the actual application.– Need to involve users in knowledge acquisition and
engineering tasks?• Incentives are related to organizational and social factors.• Seamless integration of new features.
• Semantic tools– Game mechanics.– Paid crowdsourcing (integrated).
• Using results of casual games.
http://gapingvoid.com/2011/06/07/pixie-dust-the-mountain-of-mediocrity/
Case studies
• Methods applied– Mechanism design– Participatory design– Games with a purpose– Crowdsourcing via MTurk
• Semantic contentauthoring scenarios– Extending and populating
an ontology– Aligning two ontologies– Annotation of text, media
and Web APIs
Mechanism design in practice
• Identify a set of games that represents your situation.• See recommendations in the literature.
• Translate what economists do into concrete scenarios.• Assure that the economists’ proposals fit to the concrete situation.
• Run user and field experiments. Results influence HCI, social and data management aspects.
8/27/2011 www.insemtives.eu 15
Factors affecting mechanism design
8/27/2011 www.insemtives.eu 16
Goal Tasks Social Structure
Nature of good being produced
Communication level (about the
goal of the tasks)
High
Variety of
High
Hierarchy neutral
Private goodMedium Medium
Low Low
Participation level (in the definition
of the goal)
High
Specificity of
High
Public goodMedium Medium
Low Low
Clarity levelHigh Identification
withHigh
HierarchicalCommon resourceLow
Low Required skillsHighly specific
Club goodTrivial/Common
More at http://www.insemtives.eu/deliverables/INSEMTIVES_D1.3.1.pdf andhttp://www.insemtives.eu/deliverables/INSEMTIVES_D1.3.1.pdf
Mechanism design for Telefonica
• Interplay of two alternative games– Principal agent game
• The management wants employees to do a certain action but does not have tools to check whether employees perform their best effort.
• Various mechanisms can be used to align employees’ and employers’interests
– Piece rate wages (labour intensive tasks)– Performance measurement (all levels of tasks)– Tournaments (internal labour market)
– Public goods• Semantic content creation is non-rival and non-excludable• The problem of free riding
• Additional problem: what is the optimal time and effort for employees to dedicate to annotation
4/14/11 www.insemtives.eu 18
Mechanism design for Telefonica (ii)
• Principal agent game– Pay-per-performance
• Points assigned for each contribution
– Quality of performance measurement
• Rate user contributions• Assign quality reviewers
– Tournament• Visibility of contributions by
single users• Search for an expert based on
contributions• Relative standing compared to
other users
• Public goods game– To let users know that their
contribution was valuable– The portal should be useful
• Possibility to search experts, documents, etc.
• Possibility to form groups of users and share contributions
– The portal should be easy to use
• Experiments– Pay-per-tag vs winner-takes-
it-all for annotation.
4/14/11 www.insemtives.eu 19
Knowledge engineering tasks• Granularity of ontology
engineering activities is toobroad; further splitting isneeded
• Crowdsource very specifictasks that are (highly) divisible– Labeling (in different
languages)– Finding relationships– Populating the ontology– Aligning and interlinking– Ontology-based annotation– Validating the results of
automatic methods– …
www.insemtives.eu 20
OntoGame API• API that provides several methods that are
shared by the OntoGame games, such as: – Different agreement types (e.g. selection
agreement).– Input matching (e.g. , majority).– Game modes (multi-player, single player). – Player reliability evaluation. – Player matching (e.g., finding the optimal
partner to play).– Resource (i.e., data needed for games)
management.– Creating semantic content.
• http://insemtives.svn.sourceforge.net/viewvc/insemtives/generic-gaming-toolkit
www.insemtives.eu 218/27/2011
Lessons learned• Tasks which can be subject to games
– Definition of vocabulary– Conceptualization
• Based on competency questions• Identifying instances, classes, attributes, relationships
– Documentation• Labeling and definitions• Localization
– Evaluation and quality assurance• Matching conceptualization to documentation
– Alignment– Validating the results of automatic methods
• But, the approach is per design less applicable because– Knowledge-intensive tasks that are not easily nestable– Repetitive tasks players‘ retention?
www.insemtives.eu 24
Lessons learned (ii)• Approach is feasible for mainstream domains, where a
knowledge corpus is available• Knowledge corpus has to be large-enough to allow for
a rich game experience– But you need a critical mass of players to validate the
results• Advertisement is essential• Game design vs useful content
– Reusing well-kwown game paradigms– Reusing game outcomes and integration in existing
workflows and tools• Cost-benefit analysis