Gathering Background Knowledge for Story Understanding through Crowdsourcing

26
Gathering Background Knowledge for Story Understanding through Crowdsourcing Christos T. Rodosthenous & Loizos Michael Computational Cognition Lab, Open University of Cyprus

description

Successfully comprehending stories involves integration of the story information with the reader's own background knowledge. A prerequisite, then, of building automated story understanding systems is the availability of such background knowledge. We take the approach that knowledge appropriate for story understanding can be gathered by sourcing the task to the crowd. Our methodology centers on breaking this task into a sequence of more specific tasks, so that human participants not only identify relevant knowledge, but also convert it into a machine-readable form, generalize it, and evaluate its appropriateness. These individual tasks are presented to human participants as missions in an online game, offering them, in this manner, an incentive for their participation. We report on an initial deployment of the game, and discuss our ongoing work for integrating the knowledge gathering task into a full-fledged story understanding engine.

Transcript of Gathering Background Knowledge for Story Understanding through Crowdsourcing

Page 1: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Gathering Background Knowledge for Story Understanding through Crowdsourcing

Christos T. Rodosthenous & Loizos Michael Computational Cognition Lab, Open University of Cyprus

Page 2: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Introduction General research problem

Acquisition of Background Knowledge to be used by an automated story understanding system

Approach to the problem Develop a method / system to facilitate knowledge acquisition using crowdsourcing techniques Source task to the crowd

Page 3: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Engine Architecture

Page 4: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Knowledge Representation Gather knowledge in a machine-readable form

High-level version of the Event Calculus (Michael, 2010)

A fluent F: is an object whose value can change through the course of time An action A: event that occurs at a specific time-point A literal L: fluent or an action, or their negation

Page 5: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Knowledge Representation Φ implies L: constraints that hold at each story time-point e.g., person(X) implies can(X,think)

Φ causes L: capture the conditions whose presence at some time-point is sufficient to change the state of L at the next time-point e.g., attack(X,Y) causes war(X,Y)

Page 6: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Crowdsourcing “A strategy that combines the effort of the public to solve a problem or produce a resource” (Wang et al., 2013). Games With A Purpose (GWAPs)

Genre of crowdsourcing ESP Game, Verbocity, Common Consensus

Page 7: Gathering Background Knowledge for Story Understanding through Crowdsourcing

“Knowledge Coder” Game Output-agreement games template

Players required to agree on the same output they produce Game story

Planet Earth is captured by alien forces capable of intercepting human communications in natural language. Join the resistance forces and encode human knowledge in a form that is not readable by aliens.

Page 8: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Scoring and incentives Players are rewarded with points:

For each successful mission attempt When other players contribute and verify the former players’ mission results and vice versa

Awards issued after a certain score

Page 9: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Considerations Assumptions for human participants

Knowledgeable Honest Willing to participate

Game comprises multiple steps

Lower probability of user error Easier control of the outcomes of each step Facilitate the integration with knowledge understanding systems

Page 10: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Considerations Cheating: non-standard methods for creating an advantage beyond normal gameplay Anti-cheating mechanisms

Player anonymity Internet address recording/filtering Time-bounded missions

Page 11: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Acquiring Knowledge Acquire broad knowledge

Use in Multiple Stories Methodology comprises 6 steps casted as game missions:

Mission 1 - Sentence processing Mission 2 - Verb and noun identification Mission 3 - Predicate construction Mission 4 - Rule construction Mission 5 - Rule generalization Mission 6 - Rule evaluation

Page 12: Gathering Background Knowledge for Story Understanding through Crowdsourcing

1. Sentence processing Sentence: A cat chased the mice.

After processing: {cat,chase,mouse}

Page 13: Gathering Background Knowledge for Story Understanding through Crowdsourcing

2.Verb & noun identification Selected phrase: {cat,chase,mouse}

After separation: {cat,mouse} are nouns

{chase} is a verb

Page 14: Gathering Background Knowledge for Story Understanding through Crowdsourcing

3.Predicate construction

Formal expression: chase(cat,mouse)

Selected words: {cat,mouse} nouns,{chase} verb

action

Page 15: Gathering Background Knowledge for Story Understanding through Crowdsourcing

4.Rule construction

Rule 1: chase(cat,mouse) causes fear(mouse,cat)

Rule 2: chase(cat,mouse) implies can(cat,run)

Formal expression: chase(cat,mouse)

Page 16: Gathering Background Knowledge for Story Understanding through Crowdsourcing

5. Rule generalization

Rule 2: cat(X) and chase(X,Y) implies can(X,run) Rule 1: chase(X,Y) implies can(X,run)

Rule: chase(cat,mouse) implies can(cat,run)

Page 17: Gathering Background Knowledge for Story Understanding through Crowdsourcing

6. Rule evaluation

Applicability

Sentence: A policeman was chasing a burglar.

Validity

Rule: chase(X,Y) implies can(X,run)

Page 18: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Rule Applicability The conditions in the body of the rule are met in the context of the selected sentence.

Sentence: A policeman was chasing a burglar.

Rule: chase(X,Y) implies can(X,run)

Body

X Y

Page 19: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Rule Validity Decide whether the head of the rule follows from the sentence.

Rule: chase(X,Y) implies can(X,run)

Head

Sentence: A policeman was chasing a burglar.

X

Page 20: Gathering Background Knowledge for Story Understanding through Crowdsourcing

5 participants Men and women 18+ >High School education

2 Stories (Aesop's fables) The Oxen and the Butchers The Doe and the Lion

Empirical Setting

Access to a test game deployment for 1 week Training on how to play

Page 21: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Empirical Results Number of generated rules: 93

Number of causality rules: 15 Number of implication rules: 78

Examples of generated rules horn(X) and assemble(X) and carry(purpose) and sharpen(X) and assemble(certain,X,carry(purpose)) implies have(ox,horns)

beast(X) and throw(Y,mouth,X) implies kill(X,Y)

Page 22: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Empirical Results Examples of generated rules

beast(X) and man(Y) and doe(Z) and exclaime(Z) and escape(Z,Y) and throw(Z,X) implies kill(X,Z)

Example of a “good” rule beast(X) and throw(Y,mouth,X) implies kill(X,Y)

Typos are common in GWAPs

Page 23: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Player Feedback Missions 1 and 2

Easy to play Informative instructions

Missions 3 and 4 Required time before players understand fully what they were expected to do Interesting

Mission 5 Not very challenging

Mission 6 Easy to play

Page 24: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Player Feedback Interesting game story Would advertise the game to their friends Proposed tablet and mobile version of the game Requested integration with social media

Page 25: Gathering Background Knowledge for Story Understanding through Crowdsourcing

Conclusions & Future Work Encouraging results in terms of the feasibility of our methodology Conduct further experiments with more stories and players Acquisition of not highly applicable rules

Need for stronger incentives to simplify the rules Integrate “Knowledge Coder” with a reasoning engine

Framework based on psychologically-validated models of narrative comprehension (Diakidoy et al., 2014)

Compare crowdsourcing methods used in “Knowledge Coder” game with automated knowledge acquisition methods

Page 26: Gathering Background Knowledge for Story Understanding through Crowdsourcing

More information…. • Christos T. Rodosthenous • Email: [email protected] • WWW: http://cognition.ouc.ac.cy

Game is accessible online at: http://cognition.ouc.ac.cy/narrative