Jonathan Gratch USC Institute for Creative Technologies Joint work with Stacy Marsella USC...

40
Jonathan Gratch USC Institute for Creative Technologies Joint work with Stacy Marsella USC Information Sciences Institute The Architectural Role of Emotions in Cognitive Systems

Transcript of Jonathan Gratch USC Institute for Creative Technologies Joint work with Stacy Marsella USC...

Jonathan GratchUSC Institute for Creative Technologies

Joint work with

Stacy MarsellaUSC Information Sciences Institute

Jonathan GratchUSC Institute for Creative Technologies

Joint work with

Stacy MarsellaUSC Information Sciences Institute

The Architectural Role of Emotions in Cognitive Systems

The Architectural Role of Emotions in Cognitive Systems

Outline

Emotions are adaptiveCan inform cognitive system design

Ground in implemented cognitive system– Mission Rehearsal Exercise system– Cognitive Appraisal Theory– Illustrate impact on architecture design

General implications for cognitive systems

Outline

Outline

• Revolutionary progress in emotion research– Neurophysiology of emotion (Damasio, LeDoux)– Appraisal theories (Frijda, Lazarus, Scherer)

• Emotions appear adaptive (in moderation?)– Decision-making ─ Focus of attention– Learning ─ Social relationships – Belief formation ─ Communication

• Growing interest in “emotional” systems with focus on modeling human behavior– HCI (non-verbal recognition and generation) – User and human behavior modeling– Believability/Entertainment

                  

Adaptive Role of Emotions

Outline

• Can inform intelligent behavior in general• Motivate behavior• Balancing competing goals• Balancing reaction and deliberation• Disambiguating stimuli in light of existing beliefs and commitments

• Abstract and formalize as information processing– Not new idea:

• Simon(1967), Oatley&Johson-Laird(1987), Sloman

– Revisit in light of new findings• In intelligent systems• In theories of emotion

Architectural Perspective

Grounding: Virtual Humans

Face-to-face interaction

Verbal & non-verbal behaviorSwartout, Gratch, Hill, Hovy, Johnson,

Marsella, Narayanan, Rickel, Traum, …

Marsella, Johnson & Labore

Mission Rehearsal Exercise

Social Training Simulation• Explore high-stakes social interactions in safety of VR

Mission Rehearsal Exercise

• Team decision-making in crisis situations:– Non-scripted real-time interactions

– Planning, replanning, and plan execution

– Teamwork, distributed authority and responsibility

– Collaborative, mixed initiative dialogue

– Multi-party conversations

– Verbal and non-verbal communication

– Emotionally-biased behavior

Mission Rehearsal Exercise

• Assumptions/Limitations:– Tightly focused task-related dialogue

– Near-expert decision makers

– Stylized vocabulary (military speak)

– Stylized virtual environment

MRE: Leadership Training

Speech Recognition (HTK)

Semantic Parser

Motion/ Gesture Scheduler (Beat)

Text to Speech (Festival)

World Simulator

Animation System

BDI

Haptek

Com

mun

icat

ion

Bus

Audio (Protools)

Voice Input

Vega

Projection System

Speakers (10.2)

Soar Planning

DialogueAction Selection

Per

cept

ion

NLG

Emotion

NLU pragmatics

Child Healthy:False

AccidentIntend: FalseBlame: unresolved

Assist Eagle 1-6:False

Eagle 1-6 AssistDesire: LT

Belief: False

Child-HealthyDesire: SGTBelief: False

Probability: 75%

Get MedevacResponsibility:LTIntend: True

Medevac Available:True

Past FuturePresent

Cognitive Representation

Soar’s Working Memory

Planning Perception Dialogue Action

Soar operators

Child Healthy:False

AccidentIntend: FalseBlame: unresolved

Assist Eagle 1-6:False

Eagle 1-6 AssistDesire: LT(+50)

Belief: False

Child-HealthyDesire: SGT(+80)

Belief: FalseProbability: 75%

Get MedevacResponsibility:LTIntend: True

Medevac Available:True

Past Events Future PlansPresent

Cognitive Representation

• Causal Interpretation– Combines decision-theoretic plans with models of

belief and intention– Uniform representation of past, present, future– Agent centric subjective view

Architectural Role of Emotion

• Began with view “emotion as veneer” Ended up as central organizing construct – Initial problem:

• how to convey emotion in interactive setting?

– Built mechanism to infer plausible emotions• In response to simulation events • In response to user interventions

– But discovered resolved architectural issues• Coherence is more than skin deep• Build it and they will come

How to convey emotion

• Cognitive Appraisal Theory– Influential and well-established theory

Arnold, Frijda; Lazarus; Ortony, Clore & Collins; Scherer; Smith

– Emphasizes tight coupling between• Emotion

• Cognition

• Motivation

Goals, BeliefsExternal Events

Coping

Emotion

Appraisal

Problem-focused

Emotion-focused

Cognitive Appraisal Theory

Coping

Smith and Lazarus’ cognitive-motivational-emotive system

Goals, BeliefsExternal Events Appraisal

Appraisal

• Appraisal = Situation assessment

– Compare beliefs, desires and intentions with

external circumstances

Appraisal

• Characterize via appraisal variables– Desirability– Likelihood – Urgency– Unexpectedness– Causal attribution (causality, agency, blame/credit)– Coping potential (controllability, adaptability)

• Superset of criteria considered by cog systems– Decision theory: desirability, likelihood– Scheduling: desirability, urgency

CopingProblem-focused

Emotion-focused

• Coping = Response strategy– Characterized by ontology of coping strategies

Emotion

Coping Strategies

Goals, BeliefsExternal Events

• Problem-focused (act on the world)– Action execution

– Planning

– Seek instrumental social support

• Analogous to:– Deliberative or reactive problem solving

– Team negotiation

Coping Strategies

• Emotion-focused (act on belief)– Denial– Find silver lining– Shift blame– Distancing

• Not typically considered by cog systems systems– More than a decision (e.g. abandon current plan)

• Provides self-justification for why• Related to motivational / explanatory coherence • Leads to persistent change in behavior

Coping Strategies

Modeling Appraisal and Coping

Past Future

Planning Perception Dialogue Action

Soar’s Working Memory

Soar Operators

Modeling Appraisal and Coping

• Appraisal as plan-evaluation– Causal interpretation mediates agent-environment relationship– Define appraisal variables in terms of features of interpretation– Fast, reactive, parallel

• Coping as generalized plan criticsMap to operators that change interpretation– Problem-focused execute step, add plan step– Emotion-focused

• Denial Change belief• Find silver lining Change utilities• Shift blame Change causal attribution

Dialogue moves• Distancing Drop goal / intention

Émile: Architectural Manifestation

The Emotional Octopus

Child Healthy:False

AccidentBlame: unresolved

Assist Eagle 1-6:FalseEagle 1-6 Assist

Desire: LTSatisfied: False

Child-HealthyDesire: SGT

Satisfied: False

Distress: 80

Sgt’s Appraisal of Accident from his perspective

Perspective: Self (Sgt)Desirability: -80Likelihood: 100% Blame/Credit: unresolved

Ap

praisal

Child Healthy:False

AccidentBlame: unresolved

Assist Eagle 1-6:False

Distress: 80

Child-HealthyDesire: SGT

Satisfied: False

Distress: 80

Perspective: LieutenantDesirability: -80Certainty: 100%Blame/Credit: unresolved

Sgt’s Appraisal of Accident from Lieutenant’s Perspective

Eagle 1-6 AssistDesire: LT

Satisfied: False

Ap

praisal

Child Healthy:False

AccidentBlame: unresolved

Assist Eagle 1-6:FalseEagle 1-6 Assist

Desire: LTSatisfied: False

Child-HealthyDesire: SGT

Satisfied: False

Distress: 80

Cop

ing

Child Healthy:False

AccidentBlame: unresolved

Assist Eagle 1-6:FalseEagle 1-6 Assist

Desire: LTSatisfied: False

Distress: 80

Sgt’s OwnPerspective

Cop

ing

Child-HealthyDesire: SGT

Satisfied: False

Child Healthy:False

AccidentBlame: unresolved

Assist Eagle 1-6:FalseEagle 1-6 Assist

Desire: LTSatisfied: False

Distress: 80

Child-HealthyDesire: SGT

Satisfied: FalseProbability: 75%

Distress: 80

Cop

ing

MakeAmends

Get MedevacResponsibility:LT

Problem-Focused Coping: Form intention to help Boy

Child Healthy:False

AccidentBlame: MOM

Assist Eagle 1-6:FalseEagle 1-6 Assist

Desire: LTSatisfied: False

Distress: 80

Child-HealthyDesire: SGT

Satisfied: FalseProbability: 75%

Distress: 80

Cop

ing

MakeAmends

Get MedevacResponsibility:LT

ShiftBlame

Emotion-Focused Coping: Blame Mother

Child Healthy:False

AccidentBlame: unresolved

Assist Eagle 1-6:FalseEagle 1-6 Assist

Desire: LTSatisfied: False

Distress: 80

Child-HealthyDesire: SGT

Satisfied: FalseProbability: 75%

Distress: 80

Cop

ing

MakeAmends

Get MedevacResponsibility:LT

ShiftBlame

Personality

MRE: Leadership Training

Architectural Implications

• Emotion as central control construct– Planning (inform course-of-action selection)

– NLU (inform reference resolution)

– Dialogue (prompt dialogue initiative)

– NLG (biases sentence generation strategies)

– Non-verbal expression

General Implications

• Emotion and Reflection– Appraisal is form of self-reflection / focus of attention

• Emotion as decision-making– Generalization of decision-theory

– More to the world than probabilities and utilities

• Emotion and plausible reasoning– Emotion-focused coping motivate preference/beliefs

– Attempt to construct coherent motivational explanation

– Non-rational but adaptive?

General Implications

• Emotion and Learning– Focus learning on “emotionally salient” events

– Appraisal variables as features / case indexes

Conclusion

• Emotion is form of information processing– Arguable adaptive

• Juggling competing goals and commitments

• Focusing cognitive resources

• Enforcing coherence

– Arguable unexplored by cognitive systems

Focus=1Lt: U9 “Secure a landing zone”Committed(lt,7), 7 authorized, Obligation(sgt,U9)Sgt: U10 “First we should secure the assembly area”Disparaged(sgt, 7), endorsed(sgt,2)Lt: U11“Secure the area”Committed(lt,2), 2 authorized, Obligation(sgt,U11)Sgt: U12 “Yes sir”Committed(sgt,2), Push(2)Goal7:Announce(2,{1sldr,2sldr,3sldr,4sldr})Goal8: Start-conversation(sgt, {1sldr,2sldr,…},2)Goal8 Sgt: U13 “Squad leaders listen up!”Goal7 Sgt: U14 “I want 360 degree security”Push(3)Goal9:authorize 3Goal9 Sgt: u15“1st squad take 12-4”Committed(sgt,3), 3 authorizedPop(3), Push(4)Goal10: authorize 4Goal10 Sgt: u16“2nd squad take 4-8”Committed(sgt,4), 4 authorizedPop(4)…A10: Squads moveA10: grounds U13-U18,… ends conversation about 2, realizes 2Pop(2), Push(7)

Dialogue Example: Sgt’s Behavior

Render Aid

Secure Area

Secure 12-4

Secure 8-12 Secure Accident

Secure 4-8

Squads in area

A=Lt, R=Sgt

A=Sgt, R=1sldr

A=Sgt, R=2sldr

A=Sgt, R=4sldrA=Sgt, R=3sldr

Area Secure

1

2

34

5 6

Decomposition

DecompositionSecure LZ

A=Lt, R=Sgt

7

• Appraisal as a mediating variable– Direct mappings (e.g. Hayes-Roth personality model)

– Indirect mappings

• Direct: More links, No insight on how to map• Indirect: more constrained. More modular

More than a theory of emotion

World stateBeliefsDesiresPersonality

Behavior

World stateBeliefsDesiresPersonality

BehaviorAppraisal Variables

Mediating Variable

• Appraisal Mediates Personality

Personality Variable Appraisal Variables Behavior

e.g Extroversion Control Hope

Penley & Tomaka (2002)

• Appraisal Mediates Culture

Culture Variable Appraisal Variables Behavior

e.g. Uncertainty avoidance Threat Fear Kupperbusch et al

Mediating variables

• Coping mediated by appraisal– Undesirable & Controllable

Distress

Problem directed coping

– Undesirable & Uncontrollable

Distress

Emotion directed coping