Generation of Referring Expressions: Modeling Partner Effects Surabhi Gupta Advisor: Amanda Stent...

Generation of Generation of Referring Referring

Expressions: Expressions: Modeling Partner Modeling Partner

EffectsEffectsSurabhi GuptaSurabhi Gupta

Advisor: Amanda StentAdvisor: Amanda Stent

Department of Computer Department of Computer ScienceScience

OutlineOutline

IntroductionIntroduction DataData Previous workPrevious work Modeling partner effectsModeling partner effects Generating NP postmodifiersGenerating NP postmodifiers A little statistical experimentA little statistical experiment Discussion and Future WorkDiscussion and Future Work

Referring ExpressionsReferring Expressions A referring expression denotes (or points A referring expression denotes (or points

to) an object in the world of a discourse.to) an object in the world of a discourse. Examples of referring expressions include Examples of referring expressions include the the

red chairred chair, , the 400 dollar red chairthe 400 dollar red chair and and 5 red 5 red chairschairs. .

Referring expressions are usually noun phrasesReferring expressions are usually noun phrases Improper construction of a referring Improper construction of a referring

expression can result in expression can result in referring expressions that are ambiguous (e.g. referring expressions that are ambiguous (e.g.

the bookthe book when there are two books). when there are two books). referring expressions that are too descriptive referring expressions that are too descriptive

and lead to false implicatures (e.g. and lead to false implicatures (e.g. the 400 the 400 dollar chairdollar chair when there is only one chair) when there is only one chair)

Structure of a Noun Structure of a Noun PhrasePhrase

A definite/indefinite noun phrase is A definite/indefinite noun phrase is constructed of:constructed of: An (optional) determiner or quantifier e.g. An (optional) determiner or quantifier e.g. aa, ,

threethree A number of premodifiers (adjectives, adverbs, A number of premodifiers (adjectives, adverbs,

noun modifiers) e.g. noun modifiers) e.g. redred A number of postmodifiers (prepositional A number of postmodifiers (prepositional

phrases, relative clauses) e.g. phrases, relative clauses) e.g. worth 400 worth 400 dollarsdollars, , that is redthat is red

Other noun phrases include pronouns, Other noun phrases include pronouns, proper nouns, deicticsproper nouns, deictics

Adaptation in Adaptation in ConversationConversation

When people talk with each other, they When people talk with each other, they adapt to the other’s choice of referring adapt to the other’s choice of referring expression (Clark 1996, Levinson 1983, expression (Clark 1996, Levinson 1983, Brennan 1987).Brennan 1987).

Example:Example: (A): Lets buy (A): Lets buy the 400 dollar red chairthe 400 dollar red chair (B): That’s a good idea. The (B): That’s a good idea. The chairchair matches matches

with the red table.with the red table. (A): The (A): The chair chair it is then.it is then.

Generation of Referring Generation of Referring Expressions in DialogExpressions in Dialog

When a computer constructs human When a computer constructs human language, it is called language, it is called generationgeneration NewsBlaster summaries, or Google NewsBlaster summaries, or Google

translationtranslation Generation for dialog must involve Generation for dialog must involve

consideration of the dialog partner consideration of the dialog partner (the human)(the human)

Good Generation of Good Generation of Referring ExpressionsReferring Expressions

The algorithm should generate a referring The algorithm should generate a referring expression for which the human reader can expression for which the human reader can identify the referent.identify the referent.

The algorithm should generate referring The algorithm should generate referring expressions that do not lead the human reader to expressions that do not lead the human reader to make false implicatures (Grice 1968).make false implicatures (Grice 1968).

The algorithm should model how conversational The algorithm should model how conversational partners adapt to each other.partners adapt to each other.

The algorithm should be able to generate the The algorithm should be able to generate the whole range of referring expressions observed in whole range of referring expressions observed in discourse.discourse.

The algorithm should be computationally feasible.The algorithm should be computationally feasible.

Our ObjectiveOur Objective We are building a model of referring We are building a model of referring

expression generation that captures expression generation that captures adaptation to partners in adaptation to partners in conversation.conversation.

Related work in this field does not Related work in this field does not include partner adaptation for dialog include partner adaptation for dialog (Dale and Reiter 1995, Siddharthan (Dale and Reiter 1995, Siddharthan and Copestake 2004).and Copestake 2004).

OutlineOutline


DataData Two corpora of spoken dialog rich in noun phrases:Two corpora of spoken dialog rich in noun phrases:

Maptask - Speaker A giving instructions to Maptask - Speaker A giving instructions to Speaker B about following directions in a mapSpeaker B about following directions in a map

Coconut - Two participants trying to buy furniture Coconut - Two participants trying to buy furniture by using both of their inventories and money.by using both of their inventories and money.

For each corpus, we:For each corpus, we: Automatically extracted the noun phrasesAutomatically extracted the noun phrases Annotated the noun phrases by hand for referent Annotated the noun phrases by hand for referent

(in a knowledge representation we built), type (in a knowledge representation we built), type (noun phrase or pronoun), and to indicate (noun phrase or pronoun), and to indicate whether the noun phrase was embedded in whether the noun phrase was embedded in another noun phrase. another noun phrase.

CoconutCoconut MaptaskMaptask

DefDef 116116 21182118

IndefIndef 967967 14111411

1st person pronoun1st person pronoun 440440 563563

2nd person pronoun2nd person pronoun 165165 12751275

3rd person pronoun3rd person pronoun 7979 614614

DeicticsDeictics 00 00

Proper NounsProper Nouns 00 00

Quantity NounsQuantity Nouns 291291 160160

Mass NounsMass Nouns 00 00

No ModifiersNo Modifiers 1313 113113

Not EmbeddedNot Embedded 229229 16331633

EmbeddedEmbedded 242242 2626

Set ConstructionsSet Constructions 00 00

Not in KRNot in KR 612612 18751875

NPs UsedNPs Used 471471 12941294

TotalTotal 17671767 59865986

OutlineOutline


Algorithms ComparedAlgorithms Compared Rule BasedRule Based

Dale and Reiter 1995Dale and Reiter 1995 With partner effects (x 2)With partner effects (x 2) With postmodifier ordering (x 4)With postmodifier ordering (x 4)

Siddharthan and Copestake 2004Siddharthan and Copestake 2004 With partner effects (x 2)With partner effects (x 2) With postmodifier ordering (x 4)With postmodifier ordering (x 4)

StatisticalStatistical Support Vector MachinesSupport Vector Machines

Rule-Based AlgorithmsRule-Based Algorithms

Terms used:Terms used: Contrast Set:Contrast Set: contains information of all contains information of all

the objects in the world.the objects in the world. Preferred list of attributes:Preferred list of attributes: the attributes the attributes

that are known for the objects. that are known for the objects. For Coconut: type, quantity, cost, color, stateFor Coconut: type, quantity, cost, color, state

E.g. three green high tables worth $400E.g. three green high tables worth $400 Intended Referent:Intended Referent: The object from the The object from the

world, which we are trying to describe.world, which we are trying to describe.

Dale and ReiterDale and Reiter

Basic idea:Basic idea: Specify the preference list by handSpecify the preference list by hand Repeat until all members of the Repeat until all members of the

contrast set are gone:contrast set are gone: Add the value for the next attribute from Add the value for the next attribute from

the preference list for the intended the preference list for the intended referent to the noun phrase to be referent to the noun phrase to be generatedgenerated

Example:Example:Preference list: {Type, Color, Cost, Quantity, Preference list: {Type, Color, Cost, Quantity, State}State}Contrast set: {Contrast set: {300 dollar red couch, 200 300 dollar red couch, 200 dollar green couch, 250 dollar brown tabledollar green couch, 250 dollar brown table } }Intended referent: Intended referent: 200 dollar green couch200 dollar green couchGenerated NP: Generated NP: green couchgreen couch

300 dollar red couch

200 dollar green couch

250 dollar brown table

Siddharthan and Siddharthan and CopestakeCopestake

Basic idea: See Dale and ReiterBasic idea: See Dale and Reiter Preference list is reordered by using Preference list is reordered by using

synonyms and antonyms of words in synonyms and antonyms of words in each attribute each attribute

Benefits to Rule Based Benefits to Rule Based AlgorithmsAlgorithms

They consider the way humans actually converse They consider the way humans actually converse ie. humans use unnecessary attributes, they also ie. humans use unnecessary attributes, they also begin mentioning a referring expression without begin mentioning a referring expression without scanning the entire list of distractors.scanning the entire list of distractors.

They do not attempt to look for the optimal They do not attempt to look for the optimal number of attributes. They just go through the number of attributes. They just go through the list of preferred attributes and iteratively list of preferred attributes and iteratively includes those attributes that rule out at least includes those attributes that rule out at least one distractor from the contrast set.one distractor from the contrast set.

There is no backtracking and the head noun is There is no backtracking and the head noun is always included.always included.

Disadvantages to Rule Disadvantages to Rule Based AlgorithmsBased Algorithms

They don’t generate the whole range of They don’t generate the whole range of referring expressionsreferring expressions Ones with postmodifiersOnes with postmodifiers PronounsPronouns DeicticsDeictics

They don’t model adaptation to They don’t model adaptation to partnerspartners..

OutlineOutline


Adding Partner EffectsAdding Partner Effects

A rule based algorithmA rule based algorithm Basic idea: See Dale and Reiter, Basic idea: See Dale and Reiter,

Siddharthan and CopestakeSiddharthan and Copestake Preference list is reordered to match Preference list is reordered to match

selection of attributes in previous selection of attributes in previous mentions of the intended referent.mentions of the intended referent.

Variant to this where those attributes Variant to this where those attributes mentioned previously are definitely mentioned previously are definitely included even if all the competitors have included even if all the competitors have been eliminated.been eliminated.

EvaluationEvaluation

Metric: Correct / Correct + Inserted Metric: Correct / Correct + Inserted + Deleted + Moved+ Deleted + Moved

Example:Example: Human: the big fat green catHuman: the big fat green cat Computer: the green happy catComputer: the green happy cat

Correct: the, catCorrect: the, cat Inserted: happyInserted: happy Deleted: big, fatDeleted: big, fat Moved: greenMoved: green Score: 2 / 6Score: 2 / 6

ResultsResults

The variant to our partner effects The variant to our partner effects algorithms performs significantly algorithms performs significantly better that our Baseline, Dale and better that our Baseline, Dale and Reiter and Siddharthan and Reiter and Siddharthan and Copestake for both the cropora Copestake for both the cropora used.used.

OutlineOutline


Discussion and Discussion and ConclusionsConclusions

The corpus you choose makes a The corpus you choose makes a differencedifference Maptask: Few distractors, no Maptask: Few distractors, no

significant different between Baseline, significant different between Baseline, Dale and Reiter and Siddharthan and Dale and Reiter and Siddharthan and CopestakeCopestake

Do partner effects make a Do partner effects make a difference?difference?

ReferencesReferences Advaith Siddharthan and Ann Copestake. 2004. Advaith Siddharthan and Ann Copestake. 2004.

Generating Referring Expressions in Open Domains. Generating Referring Expressions in Open Domains. In In Proceedings of the 42th Meeting of the Association Proceedings of the 42th Meeting of the Association for Computational Linguistics Annual Conference for Computational Linguistics Annual Conference (ACL 2004),(ACL 2004), Barcelona, Spain. Barcelona, Spain.

Grice, H P (1975). Logic and conversation. In P. Cole Grice, H P (1975). Logic and conversation. In P. Cole and J. Morgan, editors, and J. Morgan, editors, Syntax and SemanticsSyntax and Semantics: Vol 3, : Vol 3, Speech ActsSpeech Acts, pages 43-58. New York: Academic Press., pages 43-58. New York: Academic Press.

Grosz, B and Sidner, C (1986). Attention, intention, Grosz, B and Sidner, C (1986). Attention, intention, and the structure of discourse. and the structure of discourse. Computational Computational LinguisticsLinguistics, 12: 175-206., 12: 175-206.

Robert Dale and Ehud Reiter. 1995. Computational Robert Dale and Ehud Reiter. 1995. Computational interpretations of the Gricean maxims in the interpretations of the Gricean maxims in the generation of referring expressions. Cognitive generation of referring expressions. Cognitive Science, 19:233–263.Science, 19:233–263.

AcknowledgementsAcknowledgements

Dr. Amanda Stent, for all her time and Dr. Amanda Stent, for all her time and efforts during the last three years.efforts during the last three years.

The Natural Language Processing Lab The Natural Language Processing Lab in Computer Science.in Computer Science.

The Honors College for giving me the The Honors College for giving me the chance of working on this year long chance of working on this year long project.project.

NSFNSF

Questions?Questions?

OutlineOutline


Generating with Generating with PostmodifiersPostmodifiers

Why? -- because previous algorithms don’t Why? -- because previous algorithms don’t but it’s a big part of the corpus we have but it’s a big part of the corpus we have used.used.

Random - randomly decide whether the Random - randomly decide whether the attribute selected should be a post attribute selected should be a post modifier or premodifiermodifier or premodifier

Unigrams - see where the attribute is in Unigrams - see where the attribute is in relation to the ‘type’.relation to the ‘type’.

Bigrams - statistics of pairs of attributes. Bigrams - statistics of pairs of attributes. E.g probability of finding an attribute E.g probability of finding an attribute given another.given another.

ResultsResults

OutlineOutline


Support Vector MachinesSupport Vector Machines SVMs are a set of machine learning SVMs are a set of machine learning

algorithms for binary classification that algorithms for binary classification that have been applied to NLP.have been applied to NLP.

We used a set of SVMs, one per attribute, We used a set of SVMs, one per attribute, that voted “yes” or “no” to use this that voted “yes” or “no” to use this attribute at this point in the noun phrase.attribute at this point in the noun phrase. Maptask: 6 attributes, Coconut: 5 attributesMaptask: 6 attributes, Coconut: 5 attributes

We evaluated using:We evaluated using: 10-fold cross-validation for Maptask.10-fold cross-validation for Maptask. 4-fold cross-validation for Coconut.4-fold cross-validation for Coconut.

EvaluationEvaluation

ResultsResults

Generation of Referring Expressions: Modeling Partner Effects Surabhi Gupta Advisor: Amanda Stent...

Documents

Transcript of Generation of Referring Expressions: Modeling Partner Effects Surabhi Gupta Advisor: Amanda Stent...