Dialogue ManagerSenior Companion

University of Sheffield

March 2008

review

Senior Companion Dialogue Manager 2-tier planning where

High-level plans built using the Cognitive Model Topic specific plans built using Dialogue Action Forms

DAFS Dialogue Action Forms Augmented Transition networks Represent domain knowledge Modularizable to encapsulate sub-topics within a conversation

domain

Back channel response while working on more complete system utterances (hm, uh-huh)

DialogueManager

Natural Language

Understanding

Dialoguehistory

Ontology Photoapp

Generation

NLU output

of user utterance

Semantic content of

system utterance

+

emotion tags

CognitiveModel

Dialogue Manager Architecture

DialogueManager

•Dialogue Acts•Named Entities

•Person•Family Relations

•Sister•Mother•grandmother

•Location•Celebration event

•Birthday•Anniversary•Wedding•Christmas

•Pronoun references•Semantic categories for

•User input objects (nouns)•User input actions(verbs)

•Original user input string•Modalities for user input

NLP output•Emotion Tags•Planning•System Model

Cognitive Model

•Hierarchy of Semantic tags for Photos•User instances

Ontology

•Record of User and systemutterances

•Photo/s id•Photo tags•Photo attributes

Photo App

DialogueHistory

DM input closeup

DialogueManager

•Number of output Utterancesfor each:•Semantic Representation

•objects•actions

•Dialogue Act•Emotional content•Output modality

•Expected user response category

Cognitive Model

Fission/Generation

Speech Analysis

Dialogue History

OntologyPhoto App

DM output closeup

Dialogue Action Forms

GUI editor for creating DAFsComposed of nodes and arcs containing

tests and actionsDAFs pre-stacked, but can be overidden

by matching indexing terms (semantic classes, significant words)Essential for mixed-initiative conversation

SC Dialogue Manager Stack

goodbye

greeting

Run Greeting DAFPop greeting DAFPush photo DAF

goodbye

photo

System start

goodbye

location

event

people

date

special memory

goodbye

location

event

date

special memory

Run photo DAFRun people DAFPop people DAF

People DAF

DAFs (1)

DAFs(2)

SC Output (1) Natural Language Generation

Currently choosing from a selection of pre-defined surface forms. A Generation module to be developed (CU)

To be shared with the Health and Fitness Companion To include input from the Cognitive model for emotion and

conversational style (based on user profile) Will contain lists of alternative question realisations derived from

corpora

TTS Loquendo TTS

Avatar

AAA, CrazyTalk, Nabaztag (2-D)

more detail

Current Dialogue Manager Architecture

InputQueue

Input Manager

NLU Photo Watcher

DAF EngineDAFStack

OutputQueue

Output Manager NLG

Output of ASRIn the format of Galaxy Communicator frame.For example:{ c output

:hypothesises ( {c hypothesis …} {c hypothesis …} … ) }

{c hypothesis:ROName “Language Model Name”:Likelyhood -159.035736:Confidence 0.758829:String “nice one”:NumOfWords 2:WordsInfo: ( {c word …} {c word …} ) }

{c word:String “nice”:AcousticScore 3.791455:Confidence 0.750377:StartPoint 0:EndPoint 42:Language: “en-gb”}

NLU

InputA textual string. For example: “Kate is the mother of Marry”.

Output1. GATE Annotation Set.A snippet output for the sentence: “Kate is the mother of Mary.”[AnnotationImpl: id=0; type=Token; features={category=NNP, kind=word, orth=upperInitial, length=4, string=Kate}; start=NodeImpl: id=0; offset=0; end=NodeImpl: id=1; offset=4, AnnotationImpl: id=1; type=SpaceToken; features={kind=space, length=1, string= }; start=NodeImpl: id=1; offset=4;end=NodeImpl: id=2; offset=5, AnnotationImpl: id=2; type=Token; features={category=VBZ, kind=word, orth=lowercase, length=2, string=is}; start=NodeImpl: id=2; offset=5; end=NodeImpl: id=3; offset=7, AnnotationImpl: id=3; type=SpaceToken; features={kind=space, length=1, string= }; start=NodeImpl: id=3; offset=7; end=NodeImpl: id=4; offset=8……]

NLU

2. A populated ontology in RDF format

A snippet output for the sentence: “Kate is the mother of Mary.”……………………… <owl:FunctionalProperty rdf:ID="hasFather"> <rdf:type rdf:resource="http://www.w3.org/2002/07/owl#ObjectProperty"/> <rdfs:range rdf:resource="#Man"/> <rdfs:domain rdf:resource="#Person"/> <rdfs:subPropertyOf rdf:resource="#hasParent"/> </owl:FunctionalProperty> <Person rdf:ID="someone2"> <hasMother> <Person rdf:ID="someone1"> <hasName rdf:datatype="http://www.w3.org/2001/XMLSchema#string" >Kate</hasName> <hasSex rdf:resource="#female"/> </Person> </hasMother> <hasName rdf:datatype="http://www.w3.org/2001/XMLSchema#string" >someone2</hasName> </Person></rdf:RDF>

DAFs

DAFs are either triggered byBeing on the top of the stack Through index term matching

Each DAF has an associated set of terms associated with it (semantic categories)

Dialogue Manager Strategy

Very simple for now, discusses variety of things in and about the photos People - their names, ages and relationship to the

user Location of the photo and when it was taken and

for what purpose

DM strategy at present connects all photos that contain the same person and discusses the similarities and differences.

DAFs

Created using a GUI editor

Set of nodes and arcsA node represents a dialogue stateEach arc contains a test and an associated

action

DAF Editor

Each node represents a dialogue state

Each arc contains a test with an associated action

The main knowledge base will be an ontology which will hold all the world knowledge. There is another ontology which holds the current session knowledge. We chose ontologies to represent the knowledge because in the same file we have a structure, instances and logical rules.

The tests invoke rules predefined in the ontology. If the rule evaluates to true, then the relative action is invoked, otherwise not. Like this, a person creating the rule needs not know anything on programming.

The rules will look like ...

Person(?p) ^ hasAge(?p,?age) ^ greaterThan(?age,200) -> TooOld(?p)

They can be written and tested in Protege using a graphical interface.

DAF tests invoke protégé Rules

SWRL http://protege.cim3.net/cgi-bin/wiki.pl?SWRLLanguageFAQ

SWRL is an acronym for Semantic Web Rule Language. It is intended to be the rule language of the Semantic Web. SWRL is based on the OWL Web Ontology Language. It allows users to write rules to reason about OWL individuals and to infer new knowledge about those individuals.

•SWRL rule contains an antecedent part, which is referred to as the body, and a consequent part, which is referred to as the head.

•Both the body and head consist of positive conjunctions of atoms

•SWRL rules are written in terms of OWL classes, properties and individuals

•SWRL supports Open World Reasoning

•One of the most powerful features of SWRL is its ability to support user-defined

• built-ins. A built-in is a predicate that takes one or more arguments and evaluates

• to true if the arguments satisfy the predicate.

･ Person(?p) ^ hasAge(?p, ?age) ^ swrlb:greaterThan(?age, 17) -> Adult(?p)

Not all user input can be structured

If the NLU module cannot fit data into a structured form of the type we are expecting then the NLU module outputs an Annotation Set which has coordinates and properties for

those coordinates.

Dialogue Manager uses the Knowledge Base to

Access individual pieces of KnowledgeStatic system knowledgeDynamic user instances

Make inferences about the knowledge *relies on the knowledge being structured

NLG

InputThe input of NLG is a logical representation which has three properties: domain, type and Value (optional). To generate a sentence of requesting some general information of a photo,following representation will be sent to NLG:

Domain: PHOTOType: REQUEST_GENERALValue: EMPTY

NLG

OutputA predefined utterance from a template file. Below is a snippet of the template file.(The $ will be replaced by the value specified in the logical representation.)

………………<Template domain="PHOTO">

<Sentences type="REQUEST_AGE"><Sentence>How old is $?</Sentence><Sentence>What is $'s age?</Sentence><Sentence>What is the age of $?</Sentence><Sentence>$ is how old?</Sentence></Sentences>

<Sentences type="REQUEST_RELATION"><Sentence>How do you know $?</Sentence><Sentence>What is your relationship with $?</Sentence></Sentences>……………..

additions to the DM (near future)

Add emotions to the SC system

Incorporate a cognitive model component

Machine Learning for improving the dialogue strategy