Louis-Philippe MorencyCarnegie Mellon University

Social and EmotionalIntelligence in AI and Agents(Modeling Human Communication Dynamics)

Social Intelligent Agents

Customer Service

News reporter

Project manager

Teacher

Co-writer

Confident

Natural Computer Interaction

Customer service

News reporter

Project manager

Teacher

Co-writer

Confident

▪ Rapport

▪ Empathy

▪ Persuasion

Social

Cognitive▪ Attention

▪ Distraction

▪ Engagement

Emotion▪ Content

▪ Surprise

▪ Frustration

Human Communication Dynamics

Human Multimodal Behaviors

▪ Gestures▪ Head gestures▪ Eye gestures▪ Arm gestures

▪ Body language▪ Body posture▪ Proxemics

▪ Eye contact▪ Head gaze▪ Eye gaze

▪ Facial expressions▪ FACS action units▪ Smile, frowning

Verbal Visual

Vocal

▪ Lexicon▪ Words

▪ Syntax▪ Part-of-speech▪ Dependencies

▪ Pragmatics▪ Discourse acts

▪ Prosody▪ Intonation▪ Voice quality

▪ Vocal expressions▪ Laughter, moans

Behavioral Multimodal Interpersonal Societal

A Central Challenge:

Modeling Human Communication Dynamics

• Vocal• Visual• Verbal

50 shades of “yeah”

Automatic Sensing for Intelligent Agents

OpenFace ToolkitFreely available for research

https://github.com/TadasBaltrusaitis/OpenFace

AI Technologies for Mental Health Assessment

ClinicianReport

Patient

MultiSense

SimSensei

OR

Clinician

Sensing User’s Mental Health Behavior Markers

DAIC

0.2

0.4

0.6

0.8

Patient Reference

Distress Not-distress2 weeks1 weekToday

Not-Depressed Depressed

Smile

Tense Voice

Open Posture

Emotional Expressiveness

Not-distressed Distressed

Distress Assessment

Interview Corpus

Depressed vs Non-depressed

Smile Dynamics - Behavior Indicators

Number of smiles

1

Smile duration

Smile intensity

S. Scherer, G. Stratou, J. Boberg, J. Gratch, A. Rizzo and L.-P. Morency. Automatic Behavior Descriptors for Psychological Disorder Analysis. IEEE Conference on Automatic Face and Gesture Recognition, 2013

PTSD vs Non-PTSD

Negative Expressions - Behavior Indicators

Overall population

2

Men only

Women only

G. Stratou, S. Scherer, J. Gratch and L.-P. Morency. Automatic Nonverbal Behavior Indicators of Depression and PTSD: Exploring Gender Differences. International Conference on Affective Computing and Intelligent Interaction, 2013

Suicidal vs Non-suicidal

Speech Patterns - Behavior Indicators

First person pronouns(e.g., me, my, mine, I)

3

Voice tenseness

Repeater vs Non-repeater

V. Venek, S. Scherer, L.-P. Morency, A. Rizzo and J. Pestian, Adolescent Suicidal Risk Assessment in Clinician-Patient Interaction, IEEE Transactions on Affective Computing, January 2016

Unusual thoughts vs No symptom

Facial Expressivity - Behavior Indicators

With clinician

4

Alone in the room

Schizophrenia

S. Vijay, T. Baltrusaitis, L.-P. Morency, L. Pennant, D. Öngür and J. Baker, Automatic prediction of psychosis symptoms from facial expressions, CHI Computing and Mental Health Workshop, 2016

MultiSense Live Demonstration

Modeling Interpersonal Dynamics

▪ Interlocutors adapt:

▪ Lexicon (gestural and verbal)

▪ Nonverbal Behavior (facial

expressions, posture)

▪ Prosody and speech

▪ High entrainment

signifies:▪ Understanding

▪ Flow of the conversation

▪ Cooperation

Interpersonal

Prediction of Immediate Negotiation Outcome

Dyadic Negotiation

Respondant’sBehaviors

Proposer’sBehaviors

JointPrediction

Model

Smile

Head Nod

Gaze

Self-touch

Smile

Head Nod

Gaze

Self-touch

History

Accept?Reject?

Predicting Listener Behaviors[IVA 2008, Best paper award]

listenerSpeaker

• Nonverbal

behaviors– Eye gaze

• Prosody

• Lexical

Prediction

Rapport Dataset

• 50 dyadic interactions

• Storytelling scenario

• Greedy forward selection

Best feature/encoding set

1. Pause

2. Eye gaze

3. “and”

4. Eye gaze

Virtual

Encodin

g

dic

tionary

• Backchannel

feedback(e.g. head nods

Latent Mixture of Discriminative Experts

Speaker

• Nonverbal

behaviors– Eye gaze

• Prosody

• Lexical

Encodin

g

dic

tionary

listener

• Backchannel

feedback(e.g. head nods

Virtual

y5y4y3y2y1

h5h4h3h2h1

x5x4x3x2x1

y5y4y3y2y1

x5x4x3x2x1

y5y4y3y2y1

x5x4x3x2x1

y5y4y3y2y1Prediction

Listeners

Discriminative experts

0

0.1

0.2

0.3

0.4

F1

me

as

ure

Rapport Dataset

Wisdom analysis

Syntax• Nouns

• Modifiers

Audio• Pauses

• Low pitch

Visual• Gaze

• Eye brows

Wisdom of

crowds

[ACL 2011, AAMAS 2010 – Best paper award]

18

MultiSense + SimSensei: Video Demonstration

Social Agents and Natural Computer Interaction

Customer service

News reporter

Project manager

Teacher

Co-writer

Confident

▪ Rapport

▪ Empathy

▪ Persuasion

Social

Cognitive▪ Attention

▪ Distraction

▪ Engagement

Emotion▪ Content

▪ Surprise

▪ Frustration

▪ Gestures▪ Head gestures

▪ Body language▪ Body posture

▪ Eye gaze

▪ Facial expressions▪ Smile, frowning

▪ Prosody▪ Voice quality

▪ Vocal expressions▪ Laughter, moans

Verbal

Visual

Vocal

MERCI !