AQUAINT Workshop – June 2003

Improved Semantic Role Parsing

Kadri Hacioglu, Sameer Pradhan, Valerie Krugler,Steven Bethard, Ashley Thornton,

Wayne Ward, Dan Jurafsky, James Martin

Center for Spoken Language Research

University of Colorado

Boulder, CO

AQUAINT Workshop – June 2003

What is Semantic Role Tagging?

• Assigning semantic labels to sentence elements.• Elements are arguments of some predicate or

participants in some event.– Who did What to Whom, How, When, Where, Why

[TEMPORAL In 1901] [THEME President William McKinley] [TARGET was shot] [AGENT by anarchist Leon Czolgosz]

[LOCATION at the Pan-American Exposition]

AQUAINT Workshop – June 2003

Parsing Algorithm

From Gildea and Jurafsky (2002)

• Generate syntactic parse of sentence (Charniak)• Specify predicate (verb)• For each constituent node in parse tree:

– Extract features relative to predicate• Path, Voice, Headword, Position, Phrase Type, Sub-Cat

– Estimate P(Role| features) for each role and normalize– Assign role with highest probability

AQUAINT Workshop – June 2003

SVM Classifier

• Same basic procedure as (Gildea & Jurafsky 2000)– Same features except include predicate as feature

• Change classification step to use SVM• TinySVM software [Kudo & Matsumoto 2000]• Prune constituents with P(Null) > 0.98

– For efficiency in training– Prunes ~ 80% of constituents

• For each role train one-vs-all classifier– Includes Null role

AQUAINT Workshop – June 2003

SVM Classification

• Generate syntactic parse (Charniak parser)

• For each target (verb)

• Prune constituents with P(Null) > 0.98

• Run each ova classifier on remaining constituents

• Convert SVM output to probs by fitting sigmoid

• Described in Platt 2000

• Generate N-best labels for each constituent

• Pick highest prob sequence of non-overlapping roles

AQUAINT Workshop – June 2003

Features

• Target word (verb)

• Cluster for target word (64)

• Path from cons to target

• Phrase Type

• Position (before/after)

• Voice

• Head Word

• Sub-categorization

Path: NP S VP VB

Head Word: He

Sub-cat: VP VB NP

AQUAINT Workshop – June 2003

Role Labels

Arg0 ArgM-ADVArg1 ArgM-CAUArg2 ArgM-DIRArg3 ArgM-DISArg4 ArgM-EXTArg5 ArgM-LOCArgA ArgM-MNRArgM ArgM-MODArgM-REC ArgM-NEGArgM-PRD ArgM-PRP

ArgM-TMP

AgentActorBeneficiaryCauseDegreeExperiencerGoalInstrumentLocationMannerMeans

PropositionResultStateStimulusSourceTemporalThemeTopicTypeOther

PropBank Arguments Thematic Roles

AQUAINT Workshop – June 2003

Data

• PropBank data– WSJ section of Penn TreeBank– Annotated with Predicate-Argument

• Train on PropBank Training Set– Section 00, 23 witheld– 72,000 annotated roles

• Test on PropBank section-23– 3,800 annotated roles

AQUAINT Workshop – June 2003

SVM Performance

Arg ID

P R F

Role

Assign

SVM 93 88 90 91

Surdeanu03 (same feat) 85 84 85 79

Surdeanu03 (add’tl feat) 92 85 89 84

Gildea & Palmer (2002) 83

Annotate PropBank ArgumentsGold-Standard Parses from TreeBank

AQUAINT Workshop – June 2003

Using Real Parses

Arg ID Role A

TreeBank Parse 93 88 90 91

Charniak Parse 88 82 85 88

Annotate PropBank Arguments

Arg ID Role A

TreeBank Parse 92 86 89 90

Charniak Parse 86 82 84 90

AnnotateThematic Roles

AQUAINT Workshop – June 2003

ID and Label

Exact FuzzyN Precision Recall Precision Recall

1 80 74 80 81

2 90 83 90 92

3 94 86 94 95

4 96 87 96 96

ID and Annotate Thematic Roles Using Charniak ParseTop N Classification

AQUAINT Workshop – June 2003

Hard vs Soft Pruning• Soft Pruning

• Train Null-vs-Role classifier on all data• Prune constituents with P(Null) > 0.98• Train ova classifiers (incl Null) on remaining constituents

• Hard Pruning• Train Null-vs-Role classifier on all data• Make Null-vs-Role classification for each constituents• Train ova classifiers (no Null) on role constituents

Soft 80 74 77

Hard 75 87 81

AQUAINT Workshop – June 2003

Segment & Classify with SVM• Initial system used Charniak parser to segment

– SVM classified segmented constituents

• Use SVM to segment and classify chunks• Features:

– Window of 5 words (+2,target,-2)

– POS tags for words

– Syntactic phrase position tags (B,I,O)

– Path from word to target

– Class assignments for previous words

• Assign Semantic phrase position tag to each word

AQUAINT Workshop – June 2003

SVM Chunking Parser

Syntactic

Parser

Path Finder

Chunker

ActivePassiveDetector

words

path for each word

POS tags

word positions

voice

Target worddetector

target word

input sentence

Features

AQUAINT Workshop – June 2003

Example I

But CC O CC<-S->VP->VBP say B A Oanalysts NNS B-NP NNS<-NP<-S->VP->VBP say B A B-agentIBM NNP B-NP VBP<-VP->SBAR->S->NP->NNP say A A B-topicis AUX O VBP<-VP->SBAR->S->VP->AUX say A A I-topica DT B-NP VBP<-VP->SBAR->S->VP->NP->DT say A A I-topicspecial JJ I-NP VBP<-VP->SBAR->S->VP->NP->JJ say A A I-topiccase NN I-NP VBP<-VP->SBAR->S->VP->NP->NN say A A I-topic

But analysts say IBM is a special case

But [AGENT analysts] [TARGET say] [TOPIC IBM is a special case]

Word POS SPP Path Pr B/A V Class

AQUAINT Workshop – June 2003

SVM Chunking Parser II

Features

POStagger

Path Finder

Yamcha

Chunker

ActivePassiveDetector

words

path for each word

POS tags

word positions

voice

Target worddetector

target word

input sentence

AQUAINT Workshop – June 2003

Example II

But_CC [NP analysts_NNS ] (VP say_VBP ) [NP IBM_NNP ] (VP is_VBZ ) [NP a_DT special_JJ case_NN ]

But CC O CC->NP->VP->VBP say B A Oanalysts NNS B-NP NNS->NP->VP->VBP say B A B-agentIBM NNP B-NP NNP->NP->VP->VBP say A A B-topicis VBZ B-VP VBZ->VP->NP->VP->VBP say A A I-topica DT B-NP DT->NP->VP->NP->VP->VBP say A A I-topicspecial JJ I-NP JJ->NP->VP->NP->VP->VBP say A A I-topic case NN I-NP NN->NP->VP->NP->VP->VBP say A A I-topic

POS tagged

& Chunked (only NP and VP)

But analysts say IBM is a special case

Word POS SPP Path Pr B/A V Class

AQUAINT Workshop – June 2003

PerformanceTrain on only first 3000 sentences PropBank data

Segment & Annotate Thematic Roles

21,000 sentences training

3000 sentences training

SVM Baseline 80/74

Chunker-1 79/71 67/53

Chunker-2 59/44

Chunker-I Syntax features derived from Charniak parse

Chunker-II Syntax features from syntactic SVM chunker

AQUAINT Workshop – June 2003

Summary and Future Work

• Project has shown continued improvement in semantic parsing

• Goals:– Improve accuracy through new features– Improve robustness to data sets by improving word

sense robustness – Continue experiments without full syntactic parse– Apply to Question Answering