Crowdsourcing ontology engineeringElena SimperlWeb and Internet Science, University of Southampton 11 April 2013

2

Overview• "online, distributed problem-

solving and production model“ [Brabham, 2008]

• Varieties: wisdom of the crowds/collective intelligence, open innovation, human computation...

• Why is it a good idea?

– Cost and efficiency savings – Wider acceptance, closer to

user needs, diversity

• Approaches

– Collaborative ontology engineering

– Challenges/competitions

– Games with a purpose – Microtask/paid

crowdsourcing

• In combination with automatic techniques

3

Crowdsourcing ontology alignment• Experiments using MTurk, CrowdFlower and established benchmarks

• Enhancing the results of automatic techniques

• Fast, accurate, cost-effective

[Sarasua, Simperl, Noy, ISWC2012]

CartP301-304

100R50PEdas-Iasted

100R50PEkaw-Iasted

100R50PCmt-Ekaw

100R50PConfOf-Ekaw

Imp301-304

PRECISION 0.53 0.8 1.0 1.0 0.93 0.73

RECALL 1.0 0.42 0.7 0.75 0.65 1.0

Open questions• Quality assurance and evaluation

• Incentives and motivators

• Choice of crowdsourcing approach and combinations of different approaches

• Reusable collection of algorithms for quality assurance, task assignment, workflow management, results consolidation etc

• Schemas recording provenance of crowdsourced data

• Descriptive framework for classification of human computation systems

– Types of tasks and their mode of execution– Participants and their roles – Interaction with system and among participants– Validation of results– Consolidation and aggregation of inputs into complete solution

Theory and practice of social machines

www.sociam.org