Assessing the Impact of Frame Semantics on Textual Entailment
description
Transcript of Assessing the Impact of Frame Semantics on Textual Entailment
![Page 1: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/1.jpg)
Assessing the Impact of Frame Semantics on Textual Entailment
Authors: Aljoscha Burchardt, Marco Pennacchiotti, Stefan Thater, Manfred Pinkal
Saarland Univ, Germanyin Natural Language Engineering 1 (1) pp1-25
As (mis-)interpreted by Peter Clark
![Page 2: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/2.jpg)
The Textual Entailment Task
Syntactically, the players have moved. from a syntactic point of view, T and H differ
But semantically, the players are still the same from a semantic point of view, T and H are the same
So, want to identify and match on semantic, not syntactic, level:
Need for “frame semantics” (syntax) X drown Y → (semantics) cause* drown victim (syntax) X drown in Y → (semantics) victim drown in cause
T: A flood drowned 11 people.H: 11 people drowned in a flood.
T: flood drown people
H: people drown in flood
* if X is inanimate (otherwise role is killer)
![Page 3: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/3.jpg)
Frame Semantic Resources
PropBank: thematic roles (arg0, arg1, …):
arg0 search arg1 for arg2 (“Mary searched the room for the ring”) arg0 search for arg2 (“Fred searched for the ring”)
BUT roles are verb-specific (and names are overloaded) arg0 seek arg1 (“Mary sought the ring”)
No guarantee arg0 means the same in different verbs
Note: thematic roles like “agent” are necessarily verb-specific: Fred sold a car to John. John bought a car from Fred. Thematic roles: Fred, John are both agents. Case/semantic roles: Fred is the buyer, John is the seller.
![Page 4: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/4.jpg)
Frame Semantic Resources
FrameNet: Semantic roles are shared among verbs
several verbs map to the same Frame Frames organized in a taxonomy Roles organized in a taxonomy
Doesn’t contain subcategorization templates for semantic role labeling causer kill victim
But does contain role-labeled examples, from which semantic role labeling algorithms can be learned
![Page 5: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/5.jpg)
Example Frame in FrameNet
![Page 6: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/6.jpg)
So: it seems FrameNet should really help!
![Page 7: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/7.jpg)
Even more, FrameNet has (limited) inferential connections
T: Wyniemko, now 54 and living in Rochester Hills, was arrested and tried in 1994 for a rape in Clinton Township.H: Wyniemko was accused of rape.
![Page 8: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/8.jpg)
But, limited success in practice
PropBank used by several systems, including the RTE3 winner: but unclear how much PropBank contributed
FrameNet used in SALSA (Burchardt and Frank) Shalmaneser + Detour for Semantic Role Labeling (SRL)
(Detour boosts SRL when training examples are missing)
SALSA: find matching semantic roles see if the role fillers match
machine learning approach: for set of known matching fillers (i) compute features (ii) learn which weighted sum of features implies match
But SALSA didn’t do significantly better than simple lexical overlap
![Page 9: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/9.jpg)
Possible reasons for “failure”
Poor coverage of FrameNet Decision of applicable Frame is poor Semantic Role Labeling is poor Role filler matching is poor
How to distinguish between these? Create FATE, an annotated RTE corpus Only annotated the “relevant” parts of the sentences
![Page 10: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/10.jpg)
FATE
Annotated RTE2 corpus (400+ve, 400-ve exs) Good interannotator agreement ~2 months work to create 4488 frames, 9512 roles annotated in the corpus
includes 373 (8%) Unknown_Frame 1% Unknown_Role → FrameNet coverage is good for this data!
Still, not always clear-cut:
Annotator: EXPORT; Shalmaneser: SENDING SENDING is still plausible
Cars exported by Japan increased
![Page 11: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/11.jpg)
![Page 12: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/12.jpg)
1. How do automatic and manual annotation compare?
Does SALSA pick the right frame?
When it picks the right frame, does assign the right
roles?
When it picks the right frame and role, does it get the right filler (i.e., the same head noun as the
gold standard)
Fred sold the book on the shelf to Mary
seller goods buyer
Commercial_Transaction
![Page 13: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/13.jpg)
Results
If H is entailed by T, then we expect1. The Frame for H to also be in T
2. The Frame’s roles used in H to also be in T
3. The role fillers in H to match those in T
These may also be true if H isn’t entailed by T BUT: presumably with less probability
![Page 14: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/14.jpg)
Results
If H is entailed by T, then we expect1. The Frame for H to also be in T (more often)
2. The Frame’s roles used in H to also be in T (more often)
3. The role fillers in H to match those in T (more often)
These may also be true if H isn’t entailed by T BUT: presumably with less probability
Also: compare with simple word overlap
![Page 15: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/15.jpg)
Results
If H is entailed by T, then we expect1. The Frame for H to also be in T (more often)
Yes…. (Note: low difference here reflects that T and H typically talk about the same thing)
![Page 16: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/16.jpg)
Results
If H is entailed by T, then we expect1. The Frame for H to also be in T (more often)
…but not much more than word overlap… (Not really surprising, as frames are picked based on words)
![Page 17: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/17.jpg)
Results
If H is entailed by T, then we expect1. The Frame for H to also be in T (more often)
Also the hierarchy doesn’t help much here
![Page 18: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/18.jpg)
Results
If H is entailed by T, then we expect1. The Frame for H to also be in T (more often)
2. The Frame’s roles used in H to also be in T (more often)
Again, low difference suggests that the roles talked about in T and H are usually the same
For pairs which have a Frame in common between T and H:
![Page 19: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/19.jpg)
Results
If H is entailed by T, then we expect1. The Frame for H to also be in T (more often)
2. The Frame’s roles used in H to also be in T (more often)
3. The role fillers in H to match those in T (more often)
![Page 20: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/20.jpg)
Results
If H is entailed by T, then we expect1. The Frame for H to also be in T (more often)
2. The Frame’s roles used in H to also be in T (more often)
3. The role fillers in H to match those in T (more often)
T: An avalanche has struck a popular skiing resort in Australia, killing at least 11 people.H: Humans died in an avalanche.
T: Virtual reality is used to train surgeons, pilots, astronauts, police officers, first-responders, and soldiers.
H: Soldiers are trained using virtual reality.
student
student
Some difficult cases:
![Page 21: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/21.jpg)
Results
Also, even if we had perfect frame, role, and filler matching, entailment does not always follow: Negation:
Modality:
![Page 22: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/22.jpg)
Conclusions
1. FrameNet’s coverage is good 2. Frame Semantic Analysis (frame/role/filler
selection) is mediocre 3. Simple lexical overlap at the frame level don’t
outperform simple lexical overlap at the syntactic level
4. Need better modeling: wider context (negation, modalities) role filler matching (semantic matching, e.g.,
WordNet) more knowledge in FrameNet, e.g., implications
e.g., kill → die, arrest → accuse
![Page 23: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/23.jpg)
(Extra slides)
![Page 24: Assessing the Impact of Frame Semantics on Textual Entailment](https://reader036.fdocuments.net/reader036/viewer/2022062422/56813de8550346895da7bf21/html5/thumbnails/24.jpg)
The Textual Entailment Task: More complex example
Again, need to match semantic roles:
Again need for “frame semantics” (syntax) X kill Y → (semantics) cause kill victim (syntax) X died in Y → (semantics) protagonist died in cause
ALSO: progagonist isa victim, Killing → Death
T: An avalanche has struck a popular skiing resort in Australia, killing at least 11 people.H: Humans died in an avalanche.
T: avalanche kill people
H: human die in avalanche