Implementation of an affective conversational agent for controlling a Hi- Fi system
description
Transcript of Implementation of an affective conversational agent for controlling a Hi- Fi system
IMPLEMENTATION OF AN AFFECTIVE CONVERSATIONAL AGENT FOR CONTROLLING A
HI-FI SYSTEM
Justo Saavedra
CONTENTS
Máster en Sistemas Electrónicos para Entornos Inteligentes
•What´s the project about? Hi-Fi
•Objectives
Introduction
•Overview
•SOAP
System Architecture
•Needs
•Appraisals
•Emotions
•Dialogue Features
Emotional System
•Classes Implemented
Object Oriented Design
•Response Generation
•Synthesizer
Hi-Fi System
•Overview of the Supervisor
Supervisor Demonstration Video
ConclusionFuture Work
Máster en Sistemas Electrónicos para Entornos Inteligentes
THE HI-FI SYSTEM
Front End
Front End
Speech Recognition
Speech Recognition
Dialogue ManagerDialogue Manager
Language Understanding
Module
Language Understanding
Module
Generation Response Module
Generation Response Module
SynthesizerSynthesizer
Execution Module
Execution Module
OBJECTIVES
Máster en Sistemas Electrónicos para Entornos Inteligentes
Improve user satisfaction when interacting with dialogue module for controlling the Hi-Fi system…
The Adaptation of a psychological model of emotions to the servant agent.
Design and development of the architectural communication system to be used.
Improvement of the existing dialogue module.
Include the new Emotional Speech Synthesizer
Máster en Sistemas Electrónicos para Entornos Inteligentes
PURPOSE
Why?
In the future emotional systems
will play an important role in the
development of intelligent/affective
systems.
Both recognition of the user emotions
(smile detection) and emotional response from the system are
highly desirable.
There is a wide field of
applications which can be benefited
from integrating an emotional system
(education, entertainment systems, social
robots, etc)
SYSTEM ARCHITECTURE
Máster en Sistemas Electrónicos para Entornos Inteligentes
Reuse, as much as possible, the previous dialogue system.
The new servers in our dialogue system, similarly to the Galaxy-II, can be in the same machine or in different machines giving broad flexibility to operate and distribute computational processing
Must provide easiness to add/delete new modules and functionalities.
SYSTEM ARCHITECTURE
Máster en Sistemas Electrónicos para Entornos Inteligentes
Main Program Loop (Hub)
Language Understanding
ModuleDialogue Manager
Response Generator
Synthesiser Client
Synthesiser server
Emotional system server &
Supervisor
Opencv Client
Opencv Webcam server
Execution Module (IR,
Roomba)
Speech recognizer
SIMPLE OBJECT ACCESS PROTOCOL
Máster en Sistemas Electrónicos para Entornos Inteligentes
SOAP is a lightweight protocol for exchange of information
Decentralized, distributed environment.
It is an XML-based portable protocol
Service RequesterEmotionClient
Service Provider EmotionServer Gets Emotions
EMOTIONAL SYSTEM
Máster en Sistemas Electrónicos para Entornos Inteligentes
Ethics
Success
Social
Safety
Survival
Adaptation of the Maslow´s Hierarchy of Needs to a robotic system
EMOTIONAL SYSTEM
Máster en Sistemas Electrónicos para Entornos Inteligentes
Ethics
Success
Social
Safety
Survival
Appraisals
Surprise Fear Happiness
Sadness Anger Shame
Neutral
Task1
Task2
Task3
EMOTIONAL SYSTEM
Máster en Sistemas Electrónicos para Entornos Inteligentes
Ethics
Success
Social
Safety
Survival
Relevance• Importance of the
situation: Going critical/Overcoming
Desirability• degree of valence of
the change provoked by the event
Surprise Fear Happiness
Sadness Anger Shame
Neutral
Task1
Task2
Task3
EMOTIONAL SYSTEM
Máster en Sistemas Electrónicos para Entornos Inteligentes
Ethics
Success
Social
Safety
Survival
Relevance
Desirability
Urgency• Estimates the time
to get to the critical level
Unexpectedness• Based on the
acceleration assuming a linear prediction model.
Surprise Fear Happiness
Sadness Anger Shame
Neutral
Task1
Task2
Task3
EMOTIONAL SYSTEM
Máster en Sistemas Electrónicos para Entornos Inteligentes
Ethics
Success
Social
Safety
Survival
Relevance
Desirability
Urgency
Unexpectedness
Unfamiliarity• Is the state a
probable situation or not
Changeability• range of the need
variation
Surprise Fear Happiness
Sadness Anger Shame
Neutral
Task1
Task2
Task3
EMOTIONAL SYSTEM
Máster en Sistemas Electrónicos para Entornos Inteligentes
Ethics
Success
Social
Safety
Survival
Relevance
Urgency
Unexpectedness
Unfamiliarity
Changeability
Controllability• Whether or not an
event is controllable depends on the capabilities
Priority (Weight)
Surprise Fear Happiness
Sadness Anger Shame
Neutral
Task1
Task2
Task3
EMOTIONAL SYSTEM
Máster en Sistemas Electrónicos para Entornos Inteligentes
Ethics
Success
Social
Safety
Survival
RelevanceDesirabilityUrgencyUnexpectednessUnfamiliarityChangeabilityControllabilityPriority (Weight)
Surprise Fear Happiness
Sadness Anger Shame
Neutral
Task1
Task2
Task3
EMOTIONAL SYSTEM
Máster en Sistemas Electrónicos para Entornos Inteligentes
Ethics
Success
Social
Safety
Survival
RelevanceDesirabilityUrgencyUnexpectednessUnfamiliarityChangeabilityControllabilityPriority (Weight)
Task1
Task2
Task3
EmotionKur urgency Ku unexpectedness Kf unfamiliarity Kagent
Kc changeability Kcont controlability
sign weight relevance Kd desirability
100
EmotionKur urgency Ku unexpectedness Kf unfamiliarity Kagent
Kc changeability Kcont controlability
sign weight relevance Kd desirability
100
SUCCESS RELATED DIALOGUE FEATURES
Máster en Sistemas Electrónicos para Entornos Inteligentes
Speech Recognition Failures
Confidence Average
Max Confidence Recognized Words Percentage
High Confidence Understood Concepts Percentage
For each phrase recognized and processed by the HiFi, it calculates several features associated with the phrase, words, concepts and actions understood.
These features are passed to the emotional system in order to calculate the success events to be generated
OBJECT ORIENTED DESIGN
Máster en Sistemas Electrónicos para Entornos Inteligentes
Several classes were implemented in order to provide code reusability and flexibility. Emotions and needs share attributes, and methods in order to calculate, and update its status
LevelTimeHistory
CIOProccess
CNeed CEmotion
OBJECT ORIENTED DESIGN
Máster en Sistemas Electrónicos para Entornos Inteligentes
• Statistics Methods (Mean, Standard Deviation)• Storage of History Levels
LevelTimeHistory
CIOProccess
CNeed CEmotion
OBJECT ORIENTED DESIGN
Máster en Sistemas Electrónicos para Entornos Inteligentes
•Process Running (Threads)
LevelTimeHistory
CIOProccess
CNeed CEmotion
OBJECT ORIENTED DESIGN
Máster en Sistemas Electrónicos para Entornos Inteligentes
CNeeds
Csurvival Csafety Csocial Csuccess Cethics
CEmotions
CAngry CShame Chappy Cfear Csad CSurprise
• Weight (Appraisals)• Critical Values• Need Formulas
• Time Constants• EmotionsUpdate(K) Formula
HI-FI SYSTEM->RESPONSE GENERATION
Máster en Sistemas Electrónicos para Entornos Inteligentes
[V_CANCI][REPRO_CD]¿Qué canción del cedé [V_DISCO] quiere?.¿Qué canción del cedé [V_DISCO] quiere escuchar?.
[V_CANCI][REPRO_CD][FRIEND]Tío, ¿Qué canción del cedé [V_DISCO] quieres?.
[V_CANCI][REPRO_CD][FRIEND][SADNESS]Perdona tío, ¿Qué canción del cedé [V_DISCO] quieres?.
Allows the user to emulate the responses by providing input parameters (template, slot, status, value, relation, emotion). Its functionality was expanded to adapt emotional behaviour
HI-FI SYSTEM->RESPONSE GENERATION
Máster en Sistemas Electrónicos para Entornos Inteligentes
Example:[V_CANCI][REPRO_CD][FRIEND][HAPPINESS]
1. Look for this template2. If not found -> Exclude the last optional parameter and look for
the new template [V_CANCI][REPRO_CD][FRIEND] 3. If not found -> Exclude the last optional parameter and look for
the new template [V_CANCI][REPRO_CD]4. If not found -> Failure
Templates:
[V_CANCI][REPRO_CD]¿Qué canción del cedé [V_DISCO] quiere?.¿Qué canción del cedé [V_DISCO] quiere escuchar?.
[V_CANCI][REPRO_CD][FRIEND]Tío, ¿Qué canción del cedé [V_DISCO] quieres?.
SYNTHESIZER
Máster en Sistemas Electrónicos para Entornos Inteligentes
A new synthesizer developed by the GTH was used. It takes into account the emotions and the intensity value.
Surprise Fear
Happiness Sadness
Neutral Anger
Shame
NegativeEmotions
PositiveEmotions
Who wins? Emotional Phrase
SYNTHESIZER
Máster en Sistemas Electrónicos para Entornos Inteligentes
A new synthesizer developed by the GTH was used. It takes into account the emotions and the intensity value.
Surprise 23.9
Fear 15.0
Happiness 25.0
Sadness 16.1
Neutral 10.0
Anger 10.0
Shame 0.0
NegativeEmotions
PositiveEmotions
Who wins? Emotional Phrase
SYNTHESIZER
Máster en Sistemas Electrónicos para Entornos Inteligentes
A new synthesizer developed by the GTH was used. It takes into account the emotions and the intensity value.
Surprise 23.9
Fear 15.0
Happiness 25.0
Sadness 16.1
Neutral 10.0
Anger 10.0
Shame 0.0
Negative Emotions = 23.9+15+16.1+10 = 65
Positive Emotions = 23.9+25 = 48.9
Dominant: SadnessIntensity: 16.1/(23.9+15.0+25.0+16.1+10+10)
Wins: Negative Emotions
SUPERVISOR
Máster en Sistemas Electrónicos para Entornos Inteligentes
Monitors/Simulate/Debug/Fine-Tune
• Monitoring of the emotions• Appraisal calculations due to certain
interaction• Decay due to time of the different
needs\emotions values
Its users interface
permits real time
monitoring
SUPERVISOR->HOW IT WORKS?
Máster en Sistemas Electrónicos para Entornos Inteligentes
• Using SOAP, the supervisor system interacts with emotional system (both send and receive data).
• The supervisor as well as the emotional system acts as a client/server depending if they are receiving or sending data.
Emotional SystemHi-Fi Supervisor
SUPERVISOR
Máster en Sistemas Electrónicos para Entornos Inteligentes
CONCLUSIONS
Máster en Sistemas Electrónicos para Entornos Inteligentes
The affective system implemented and the improvement of the dialogue response generation permits the future evaluation of the system (Compare with previous non-emotional model)
The architecture and features of the affective system (independent of task) make it possible to adapt it to other applications domains (games, tutoring systems)
The communication architecture implemented give the flexibility to change/include/delete modules with easiness
FUTURE WORK
Máster en Sistemas Electrónicos para Entornos Inteligentes
Implementing more dialog features
Fill in the necessary templates to provide better response generations
Implement more events (ethic/social)
Evaluate the system
DEMONSTRATION
Máster en Sistemas Electrónicos para Entornos Inteligentes
QUESTIONS?
Máster en Sistemas Electrónicos para Entornos Inteligentes