Pattern-Based French-to-IF AnalysisFill in the blanks IF-to-French

Generationwithin NESPOLE!

Hervé BlanchonCLIPS-IMAG

herve.blanchon@imag.fr

Outline

Introduction Analysis: Pattern-based approach

– Analyzer architecture– Steps in detail

Turn splitting into SDU Topic, sub-domain detection Argument filling DA construction

Generation– Track 1: Fill in the blanks approach– Track 2: Rule-based approach

First results Perspectives

Introduction

Handling the client's utterances

French SLT modulesFrench SLT modules

Speech RecognitionRAPHAEL

French textual output

Pattern-BasedAnalysis

IFs

Introduction

Handling the agent's utterances

French SLT modulesFrench SLT modules

Generation-Fill-in the blanks approach

-Rule-based approach

French textual output

Speech SynthesisEuler (Mons)

Sound file

IFsIFs

Hypothesis– c: c: j aimerais organiserj aimerais organiser une semaineune semaine de de

vacancesvacances dans un parc dans un parc ((I would like to I would like to organizeorganize a weeka week ofof holydaysholydays in a park in a park))

IF– {c: {c:

((

)} )}

IntroductionInterchange FormatInterchange Format

give-informationgive-information+disposition+disposition+reservation+reservation+trip+trip

Pred-partMain-predAttitudeSpeech Act

Introduction

Hypothesis– c: c: j aimerais organiserj aimerais organiser une semaineune semaine de de

vacancesvacances dans un parc dans un parc ((I would like to I would like to organizeorganize a weeka week ofof holydaysholydays in a park in a park))

IF– {c: {c:

((

)} )}

give-informationgive-information+disposition+disposition+reservation+reservation+trip+tripdisposition=( , )disposition=( , ),,

visit-spec=( , )visit-spec=( , ),,location= location= ,,duration=( , )duration=( , )

Top-level arguments

Interchange FormatInterchange Format

Introduction

Hypothesis– c: c: j aimerais organiserj aimerais organiser une semaineune semaine de de

vacancesvacances dans un parc dans un parc ((I would like to I would like to organizeorganize a weeka week ofof holydaysholydays in a park in a park))

IF– {c: {c:

((

)} )}

give-informationgive-information+disposition+disposition+reservation+reservation+trip+tripdisposition=( , )disposition=( , ),,

visit-spec=( , )visit-spec=( , ),,location= location= ,,duration=( , )duration=( , )

who=who=identifiability=identifiability=time-unit=time-unit= quantity=quantity=

Interchange FormatInterchange Format

Embedded arguments

Introduction

Hypothesis– c: c: j aimerais organiserj aimerais organiser une semaineune semaine de de

vacancesvacances dans un parc dans un parc ((I would like to I would like to organizeorganize a weeka week ofof holydaysholydays in a park in a park))

IF– {c: {c:

((

)} )}

give-informationgive-information+disposition+disposition+reservation+reservation+trip+tripdisposition=( , )disposition=( , ),,

visit-spec=( , )visit-spec=( , ),,location= location= ,,duration=( , )duration=( , )

who=who=identifiability=identifiability=time-unit=time-unit= quantity=quantity=

nonovacationvacationii likelike

weekweek 11parkpark

Interchange FormatInterchange Format

Values

Introduction

Hypothesis– c: c: j aimerais organiserj aimerais organiser une semaineune semaine de de

vacancesvacances dans un parc dans un parc ((I would like to I would like to organizeorganize a weeka week ofof holydaysholydays in a park in a park))

IF– {c: {c:

((

)} )}

give-informationgive-information+disposition+disposition+reservation+reservation+trip+tripdisposition=( , )disposition=( , ),,

visit-spec=( , )visit-spec=( , ),,location= location= ,,duration=( , )duration=( , )

who=who=identifiability=identifiability=time-unit=time-unit= quantity=quantity=

nonovacationvacationii likelike

weekweek 11parkpark

Back-translation– J'aimerais organiser des vacancesJ'aimerais organiser des vacances dans un parcdans un parc pendant une pendant une

semainesemaine. . ((I would like to organizeI would like to organize holydaysholydays in a park in a park during a weekduring a week..))

Interchange FormatInterchange Format

Analysis

Necessity to deal with– Insertions– Deletions– Wrong words & agreements

Arguments as the core of an IF– Constrain the possible ones through a topic– Find the realized ones among the possible ones

Phrase spotting mechanism– Regular expressions to express their realization– Minimal linguistic treatments

ChoiceChoice

Analysis

ASR Output Splitting into SDUs

SDU 1 SDU 2 SDU n…

Topic Detection

SDU j Topic i

Topic i handling

IFs

DA Construction

Arguments fillingArg k filling

…

…

ArchitectureArchitecture

Analysis

Simple sentences & phrases as SDU boundaries– Leading to terminal SA or SA with few continuations

AffirmAffirm– oui c'est ça, bien sûr oui c'est ça, bien sûr – yes that's it, of course– yes that's it, of course

NegateNegate – non pas du tout, non pas très bien, nonnon pas du tout, non pas très bien, non – no not at all, no not very – no not at all, no not very

well, nowell, no ExclamationExclamation

– c'est excellent, très bien, ohc'est excellent, très bien, oh – that's excellent, very good, oh – that's excellent, very good, oh GreetingGreeting

– bonjour, au revoir, bonne journéebonjour, au revoir, bonne journée – hello, good bye, have a good – hello, good bye, have a good dayday

Splitting turns into SDUSplitting turns into SDU

Splitting turns into SDUSplitting turns into SDUAnalysis

Articulations as SDU boundaries– Rhetorical + pronoun or question

ExamplesExamples– et donc jeet donc je , et puis il, et j', donc on, est-ce que, quel , et puis il, et j', donc on, est-ce que, quel

est est – and thus I, and then he, and I, so we, is …, what isand thus I, and then he, and I, so we, is …, what is

Treatment of the pronouns– When rhetorical is not explicitly stated

ExamplesExamples– xxx pronoun yyy (the|that|if) zzz pronoun tttxxx pronoun yyy (the|that|if) zzz pronoun ttt

SDU = xxx pronoun yyy (the|that|if) zzz pronoun tttSDU = xxx pronoun yyy (the|that|if) zzz pronoun ttt– xxx pronoun yyy pronoun zzzxxx pronoun yyy pronoun zzz

SDU 1 = xxx pronoun yyySDU 1 = xxx pronoun yyy pronoun zzz may have to be splitpronoun zzz may have to be split

Analysis

Goal– Find a terminal SAterminal SA or a focus conceptfocus concept– Restrict the search for instantiated

arguments Means

– Keywords are associated with each topic– Early matched topic is selected

Some topicsSome topics affirm, greeting, acknowledgment, exclamation, affirm, greeting, acknowledgment, exclamation,

please-wait, …please-wait, … accommodation, activity, attraction, package, room, accommodation, activity, attraction, package, room,

trip, …trip, …

Topic detectionTopic detection

Analysis

A function per topic (Topic2If)– Find the realized (instantiated)

arguments among the possible ones– Calculate the DA– Concatenate

The speaker The DA The arguments

Topic handlingTopic handling

Analysis

A specialized function per argument (Arg2If)– Describing the possible realizations– Capturing the pertinent information

ExempleExemple

Argument fillingArgument filling

proc RoomSpec2If {inString} {proc RoomSpec2If {inString} {

......

} elseif {[regexp } elseif {[regexp "(?:les) (\[0\-9\]+) ($fifservdico::frenchroom)(?:x|s)?""(?:les) (\[0\-9\]+) ($fifservdico::frenchroom)(?:x|s)?" $inString lMatch lQuantity lRoom]!=0} { $inString lMatch lQuantity lRoom]!=0} {

append the_result append the_result "room-spec=(identifiability=yes, quantity=" $lQuantity ", ""room-spec=(identifiability=yes, quantity=" $lQuantity ", " [fifservdico::Room2If $lRoom] ")" [fifservdico::Room2If $lRoom] ")"

} elseif {[regexp } elseif {[regexp "(\[0\-9\]+) ($frenchroom)(?:x|s)?""(\[0\-9\]+) ($frenchroom)(?:x|s)?" $inString lMatch lQuantity lRoom]!=0} { $inString lMatch lQuantity lRoom]!=0} {

append the_result append the_result "room-spec=(identifiability=no, quantity=" $lQuantity ", ""room-spec=(identifiability=no, quantity=" $lQuantity ", " [fifservdico::Room2If $lRoom] ")" [fifservdico::Room2If $lRoom] ")"

} }

......

}}

Analysis

Construct the DA using– The attitudes– The realized arguments

To get the main-predication and the pred-participants

– Other information Subject Verbal construction (give-information, request-

information, request-suggestion, suggest) Negation of the predicate …

DA constructionDA construction

Analysis

Hypothesis– c: c: j aimerais préparerj aimerais préparer une semaineune semaine de de

vacancesvacances dans un parcdans un parc # #– c: c: i would like to preparei would like to prepare a weeka week of of

vacationvacation in a parkin a park ## Domain: Domain:

– visitvisit Arguments: Arguments:

– dispositiondisposition, , visit-specvisit-spec, , durationduration, , locationlocation DA:DA:

– give-give-information+disposition+reservation+triinformation+disposition+reservation+tripp

ExempleExemple

Generation

Fill in the blanks approach (next slides) Rule based approach

– Use (parse) the IF specification files To build dictionary and grammar skeletons To label semantically each link between a terminal

symbol of the IF with its possible continuations

– Steps Associate to each terminal of the IF input the list of its

possible labelled continuations; Built a semantic tree of the IF according to the

continuations; Translate all the terminals and structures of the IF into

French words and structures; Generate a syntactically well-formed sentence in

French.

ApproachesApproaches

Generation

Step 1: Find the normalized DA– Discarding the disposition(s)

Step 2: Find the “fill in the blank sentence”– According to some of the possible

arguments Step 2: Fill in the blanks

– With the phrases generated for the arguments

Fill in the blanksFill in the blanks

Generation

IF c:c:give-information+disposition+reservation+tripgive-information+disposition+reservation+trip ((disposition=(who=i, like)disposition=(who=i, like), , visit-visit-spec=(identifiability=no, vacation)spec=(identifiability=no, vacation), ,

location=parklocation=park, , duration=(time-unit=week, quantity=1)duration=(time-unit=week, quantity=1)))

Fill in the blank sentenceifDisposition2TextifDisposition2Text organiser organiser ifVisitSpec2textifVisitSpec2text ifLocation2TextifLocation2Text ifTime2Text ifTime2Text ifDuration2TextifDuration2Text..

Result– J'aimerais J'aimerais organiserorganiser des vacances des vacances dans un parcdans un parc

pendant une semainependant une semaine..– I would like I would like to organizeto organize vacations vacations in a parkin a park during a during a

weekweek..

Fill in the blanksFill in the blanks

First results

4 unseen dialogues, clients turns– Transcription as References

2 settings– Manually segmented into SDU (all languages)– Automatically segmented into SDU (analysis

module - French only) 5 sets

– ASR alone (WER, Hypos as paraphrase of SDU)– FR-FR on ASR, FR-FR on References– FR-IT on ASR, FR-IT on References

3 graders & 3 grades– p: perfect, k: OK, b: bad

(p+k=acceptable)

EvaluationEvaluation

First results

Fr-Fr and Fr-It are comparable with Xx-Xx and Xx-It It-Fr is performing less than It-Xx

– French generator less oriented towards agent's IFs

EvaluationEvaluationMonolingual (%acceptable)

on Ref//on Rec[on auto SDU]

En-En58%//45%

Ger-Ger31%//25%

Fr-FrFr-Fr

54%/54%/41%41%[62%/[62%/48%]48%]

It-It61%//48%

Bilingual (%acceptable)

on Ref//on Rec[on auto SDU]

En-It55%//43%

Ger-It32%//27%

Fr-ItFr-It

44%/44%/34%34%[58%/[58%/44%]44%]

It-En47%//37%

It-Ger47%//31%

It-FrIt-Fr

40%/40%/27%27%

Perspectives

Pattern-based analysis– Quite promising– Analyzer for second showcase under

development more concepts (actions, attitudes, feature),

arguments (feature, focalizer, modifier, rhetorical)

Fill in the blank generation– Towards a better coverage with greater

factorization of the DA Rule-based generation

– Should be available for second showcase evaluation