Gathering Background Knowledge for Story Understanding through Crowdsourcing

download Gathering Background Knowledge for Story Understanding through Crowdsourcing

of 26

Embed Size (px)

description

Successfully comprehending stories involves integration of the story information with the reader's own background knowledge. A prerequisite, then, of building automated story understanding systems is the availability of such background knowledge. We take the approach that knowledge appropriate for story understanding can be gathered by sourcing the task to the crowd. Our methodology centers on breaking this task into a sequence of more specific tasks, so that human participants not only identify relevant knowledge, but also convert it into a machine-readable form, generalize it, and evaluate its appropriateness. These individual tasks are presented to human participants as missions in an online game, offering them, in this manner, an incentive for their participation. We report on an initial deployment of the game, and discuss our ongoing work for integrating the knowledge gathering task into a full-fledged story understanding engine.

Transcript of Gathering Background Knowledge for Story Understanding through Crowdsourcing

  • Gathering Background Knowledge for Story Understanding through Crowdsourcing Christos T. Rodosthenous & Loizos Michael Computational Cognition Lab, Open University of Cyprus
  • Introduction General research problem Acquisition of Background Knowledge to be used by an automated story understanding system Approach to the problem Develop a method / system to facilitate knowledge acquisition using crowdsourcing techniques Source task to the crowd
  • Engine Architecture
  • Knowledge Representation Gather knowledge in a machine-readable form High-level version of the Event Calculus (Michael, 2010) A fluent F: is an object whose value can change through the course of time An action A: event that occurs at a specific time- point A literal L: fluent or an action, or their negation
  • Knowledge Representation implies L: constraints that hold at each story time- point e.g., person(X) implies can(X,think) causes L: capture the conditions whose presence at some time-point is sufficient to change the state of L at the next time-point e.g., attack(X,Y) causes war(X,Y)
  • Crowdsourcing A strategy that combines the effort of the public to solve a problem or produce a resource (Wang et al., 2013). Games With A Purpose (GWAPs) Genre of crowdsourcing ESP Game, Verbocity, Common Consensus
  • Knowledge Coder Game Output-agreement games template Players required to agree on the same output they produce Game story Planet Earth is captured by alien forces capable of intercepting human communications in natural language. Join the resistance forces and encode human knowledge in a form that is not readable by aliens.
  • Scoring and incentives Players are rewarded with points: For each successful mission attempt When other players contribute and verify the former players mission results and vice versa Awards issued after a certain score
  • Considerations Assumptions for human participants Knowledgeable Honest Willing to participate Game comprises multiple steps Lower probability of user error Easier control of the outcomes of each step Facilitate the integration with knowledge understanding systems
  • Considerations Cheating: non-standard methods for creating an advantage beyond normal gameplay Anti-cheating mechanisms Player anonymity Internet address recording/filtering Time-bounded missions
  • Acquiring Knowledge Acquire broad knowledge Use in Multiple Stories Methodology comprises 6 steps casted as game missions: Mission 1 - Sentence processing Mission 2 - Verb and noun identification Mission 3 - Predicate construction Mission 4 - Rule construction Mission 5 - Rule generalization Mission 6 - Rule evaluation
  • 1. Sentence processing Sentence: A cat chased the mice. After processing: {cat,chase,mouse}
  • 2.Verb & noun identification Selected phrase: {cat,chase,mouse} After separation: {cat,mouse} are nouns {chase} is a verb
  • 3.Predicate construction Formal expression: chase(cat,mouse) Selected words: {cat,mouse} nouns,{chase} verb action
  • 4.Rule construction Rule 1: chase(cat,mouse) causes fear(mouse,cat) Rule 2: chase(cat,mouse) implies can(cat,run) Formal expression: chase(cat,mouse)
  • 5. Rule generalization Rule 2: cat(X) and chase(X,Y) implies can(X,run) Rule 1: chase(X,Y) implies can(X,run) Rule: chase(cat,mouse) implies can(cat,run)
  • 6. Rule evaluation Applicability Sentence: A policeman was chasing a burglar. Validity Rule: chase(X,Y) implies can(X,run)
  • Rule Applicability The conditions in the body of the rule are met in the context of the selected sentence. Sentence: A policeman was chasing a burglar. Rule: chase(X,Y) implies can(X,run) Body X Y
  • Rule Validity Decide whether the head of the rule follows from the sentence. Rule: chase(X,Y) implies can(X,run) Head Sentence: A policeman was chasing a burglar. X
  • 5 participants Men and women 18+ >High School education 2 Stories (Aesop's fables) The Oxen and the Butchers The Doe and the Lion Empirical Setting Access to a test game deployment for 1 week Training on how to play
  • Empirical Results Number of generated rules: 93 Number of causality rules: 15 Number of implication rules: 78 Examples of generated rules horn(X) and assemble(X) and carry(purpose) and sharpen(X) and assemble(certain,X,carry(purpose)) implies have(ox,horns) beast(X) and throw(Y,mouth,X) implies kill(X,Y)
  • Empirical Results Examples of generated rules beast(X) and man(Y) and doe(Z) and exclaime(Z) and escape(Z,Y) and throw(Z,X) implies kill(X,Z) Example of a good rule beast(X) and throw(Y,mouth,X) implies kill(X,Y) Typos are common in GWAPs
  • Player Feedback Missions 1 and 2 Easy to play Informative instructions Missions 3 and 4 Required time before players understand fully what they were expected to do Interesting Mission 5 Not very challenging Mission 6 Easy to play
  • Player Feedback Interesting game story Would advertise the game to their friends Proposed tablet and mobile version of the game Requested integration with social media
  • Conclusions & Future Work Encouraging results in terms of the feasibility of our methodology Conduct further experiments with more stories and players Acquisition of not highly applicable rules Need for stronger incentives to simplify the rules Integrate Knowledge Coder with a reasoning engine Framework based on psychologically-validated models of narrative comprehension (Diakidoy et al., 2014) Compare crowdsourcing methods used in Knowledge Coder game with automated knowledge acquisition methods
  • More information. Christos T. Rodosthenous Email: christos.rodosthenous@ouc.ac.cy WWW: http://cognition.ouc.ac.cy Game is accessible online at: http://cognition.ouc.ac.cy/narrative