CSCI 534 Affective Computing

CSCI 534(Affective Computing) – Lecture by Jonathan Gratch

CSCI 534 Affective Computing

Lecture 1: 25 Aug, 2015

Prof Jonathan Gratch


Outline

What is affective computing?

Is emotion a good thing? (for computers?)

– Review history of thinking on emotion

– Preview some applications

Structure and Goals of Course

Class projects and resources


What is “Affect”

“The conscious subjective aspect of an emotion considered

apart from bodily changes”

i.e., a conscious awareness of an emotion

This is not what we mean here!

In context of this course, “affect” refers to emotion and “related

phenomena”:

– Emotions (e.g., angry, sad, joyful, fearful)

– Moods (e.g., cheerful, gloomy, irritable, listless, depressed, buoyant)

– Interpersonal stances (e.g., distant, cold, warm, supportive)

– Preferences/Attitudes/Sentiment (e.g., liking, loving, hating)

– Personality (e.g., nervous, anxious, reckless, morose)

– Culture (e.g., Individualistic vs. Collectivist; engineering vs. social sciences)


What is Affective Computing?

An Interdisciplinary field of research

Research and develop systems that recognize, interpret, stimulate

and simulate human affect including:

– How affect influences human-computer and human-robot interaction and

machine usability

– How affective sensing can inform machine-understanding of people

– How to make computers more human-like

– The ethics of “giving” machines emotional capabilities

Covers but is not limited to the topics involving:

– Sensing and analysis (i.e., recognition of facial expressions)

– Psychology and behavior as they relate to affective computing

– Behavior generation and user interaction


But are emotions a good thing?

Affective computing presumes there is a benefit to

giving computers “emotional intelligence”

i.e., Why would we want to muck up an

intelligent machine with emotions?


Emotion a benefit or curse? A very old debate

Plato argues emotion and intellect are in opposition

Emotion

Intellect

crooked lumbering animal, … the mate of

insolence and pride, shag-eared and

deaf, hardly yielding to whip and spur.”

lover of honor and modesty and

temperance, and the follower of true glory;

he needs no touch of the whip, but is

guided by word and admonition onlyThe Allegory of the Chariot



Plato argues emotion and intellect are in opposition

Intellect

The Ideal

The Allegory of the Chariot

This is a “dual-process” model.

We’ll see these again



The Stoics were not big fans of emotion

“He only employs his passion who can make no

use of his reason”

-- Cicero

Ideal



“Reason is, and ought to be, only the slave of the passions”David Hume, 1711-1776

Maintains Plato’s notion of dual processes but

argues emotion necessary for good decisions:

• Without passions we would lack all motivation, all

impulse or drive to act, or even to reason

• Emotion = Utility

• For these reasons, many have attributed to Hume

a belief-desire model of practical reasoning, in

which our ends are given by passions (desires)






• Without passions we would lack all motivation, all

impulse or drive to act, or even to reason





Darwin’s principles

The principle of serviceable habits– Expressive “habits” that provide benefit are selected for

– Surprise is useful “so the field of vision may be increased,

and the eyeballs moved easily in any direction” pg281

– Some evolved habits may no longer be relevant for civilized man

– Sneering in rage is a sign that our ancestors fought with our teeth

whereas “civilized” humans do not



The principle of antithesis – Some expressions of emotion look the way they do because they are

the opposite of a “serviceable” expression

– Dogs “playful” expression evolved because this minimizes confusion

with their signal of anger



The principle of the direct action of the excited

nervous system on the body– Some expressions of occur because the nervous system needs to

discharge excess energy

– Laugher discharges energy associated with tension


Emotions at the dawn of Artificial Intelligence

Psychologist Ulric Neisser argued that computers

couldn’t capture human-level intelligence because

they lacked bodies and emotion

Herb Simon (a founding-father of AI) responded

that intelligent machines must have mechanisms

akin to emotion

"Motivational and emotional controls of

cognition", Psychological Review, vol. 74, 1967

Other early AI researchers also emphasized importance of

emotions for machines (Johnson-Laird; Minsky)

http://diva.library.cmu.edu/webapp/simon/item.jsp?q=/box00065/fld04976/bdl0001/doc0001/

16CSCI 534(Affective Computing) – Lecture by Jonathan Gratch

Despite early interest in emotions, AI (and even

psychology) moved on o focus on rational and dis-

embodied cognition

Chess, not “Life”

Emphasized

Logic & Rationality over emotion

Non-social phenomena

Thinking over acting

Ignored emotions

Ignored the passions

Ignored the body

But somehow, emotion was left aside

Brain in a Vat

EPIC (Meyer & Kieras97)


Emotions and AI

Even within psychology, to the extent emotion considered, it

was viewed as a distortion of good reasoning

Good reasoning is exemplified by logical and decision-theory

Focus on research that illustrates how emotions make us

stupid

Artificial Intelligence

Human Intelligence


Example: Emotional “stupidity”


Example: Emotional “stupidity”

=

=

Explanation: Losses loom larger than gains


Recent turning point: 1995

Describes “Elliot”, with brain damage to the ventromedial prefrontal

cortex. Intelligence intact as measured by standard IQ tests. Appears

normal except for unusual calm in the face of his misfortune.

However, he is completely incapable of making wise decisions; in

business and in his personal life he handles affairs disastrously.

Claim: emotion necessary for good decision-making

Occurs simultaneously with increased interest in emotion in

psychology and economics






• Without passions we would lack all motivation,

all impulse or drive to act, or even to reason





Recent turning point: 1995

Describes “Elliot”, with brain damage to the ventromedial prefrontal

cortex. Intelligence intact as measured by standard IQ tests. Appears

normal except for unusual calm in the face of his misfortune.

However, he is completely incapable of making wise decisions; in

business and in his personal life he handles affairs disastrously.

Claim: emotion necessary for good decision-making

Occurs simultaneously with increased interest in emotion in

psychology and economics

Now focus is on research that demonstrates emotions help


Example: Emotional “wisdom”

Wilson, T. D., Lisle, D., Schooler, J. W., Hodges, S. D., Klaaren, K. J., & LaFleur, S. J.

(1993). Introspecting about reasons can reduce post-choice satisfaction. Personality and

Social Psychology Bulletin, 19, 331–339.

Task: Pick a poster to take home

• One group just grabbed one they

feel good about

• Other group asked to think carefully

and write down their reasons for

choosing

• Got to reconsider 6-months later


Emergence of Affective Computing: 1995

Roz Picard introduced the term “Affective Computing” in 1995

– Her book with that title published in 1997

Originally defined as

– “computing that relates to, arises from, and deliberately

influences emotion,”


Today

A community of practice

Represented by a professional society

– Association for the Advancement of

Affective Computing

International Conference (started in 2005)

International Journal (started in 2010)


Some typical recent research topics (from TAC)

Emotion Recognition in:

– Speech: Emotion in natural speech

Depression detection

– Text Opinions in twitter; blogs

Emotocons

– Face Understanding impact of aging

Recognizing expressions with thermal

– Physiology Inferring response to music via EEG

Detecting stress from skin conductance

Synthesis– Emotional speech

– Emotional facial expressions

Games/Entertainment computing

– Responses to victory and defeat

– Affective music player

– Boredom detection

Modeling– Modeling emotional influences on decision-

making

– Modeling factors that elicit emotions

Applications

– Health – detection and shaping

– Games/entertainment – detection

and shaping; synthesis/realism

– Education – detections; shaping

Behavioral science


Current thinking: Emotions good

Emotions serve important functions, not just for

humans but any intelligent entity

Some of these functions missing from traditional

AI/rational models– Thus, and analysis of the function emotions serves in people can

improve artificial intelligences

And, regardless of their function for machines,

computers have to deal with emotional humans– Thus, they would benefit from recognizing and understanding these

processes


So what is the function emotion?


Marsella & Gratch, Journal of Cognitive Systems Research 2009

Example: Emotion “in the wild”


What is function emotion?

Physical: Shapes the body– Action preparation: energizes body, changes physical orientation

Cognitive: Shapes the mind– Rapid, continually adjusting assessment of significant events

– Interruption of behaviors and changing of goals

Social: Shapes the minds of others– Signaling: broadcast information about mental state

– Coordination: orient and coordinate group response


Emotion prepares the body for the circumstances that

elicited the emotion

Opens eyes wider: allowing in more light to

perceive information

Mouth opens and breath taken: pulling more

information to the body

When angry, blood flows to the extremities:

preparing for physical action

Surprise


Emotion shifts cognition to a mode appropriate for the

circumstances that elicited the emotion

Emotions change perception & decision-

making (e.g., Anger)

– Quicker to perceive threats (DeSteno et al 2000/2004)

– Underestimate risk (Lerner & Keltner 2000/2001)

– Use quick/heuristic reasoning

(Bodenhausen et al 1994)

– Blame others/outgroups

(Keltner et al 93; Mackie et al 00)

– Remember anger-evoking past events (Bower, 91)


Seeing emotions in others shapes our own cognition

to prepares to interact with that person

Emotions impact social interaction

– Distress elicits helping (Eisenberg et al 89)

– Anger elicits fear (Dimberg&Ohman96)

– Negotiators concede more to angry partner (van

Kleef et al. 2007)

– Emotion communicates information to other

social actors (Darwin; Parkinson01)


Emotion is also information: Mind reading

Well, Guy Cuny is the editor of the technology website, news wireless...


Practical Affective Computing

Preview of some applications of affective

computing techniques– We’ll talk about some of these in more detail later in the semester


Facial expression

recognition

Facial expression

recognition

Detect affect in

speech

Detect affect in

speech

Detect affect in

posture & gesture

Detect affect in

posture & gesture

Sentiment analysisSentiment analysis

Facial expression

synthesis

Facial expression

synthesis

Posture and

gesture synthesis

Posture and

gesture synthesis

Automatic

contingent

empathetic

feedback

Automatic

contingent

empathetic

feedback

High-level

understanding

and decision-

making

High-level

understanding

and decision-

makingEstimate prevalence

of depression

Estimate prevalence

of depression

Example: Medicine


38

Example: Entertainment and Games

Understand how to make emotionally compelling interactions

Hartholt, Gratch, Leuski, et al., At the Virtual Frontier: Introducing Gunslinger; a Multi-Character; Mixed-Reality; Story-Driven Experience. 9th International

Conference on Intelligent Virtual Agents, Amsterdam. 2009


39

Example: Skills Training

Teach us to use and control our emotions in a negotiation

David DeVault, Johnathan Mell and Jonathan Gratch. Toward Natural Turn-Taking in a Virtual Human Negotiation Agent. AAAI Spring

Symposium on Turn-taking and Coordination in Human-Machine Interaction. 2015, AAAI Press: Stanford, CA.


40

Example: Skills Training


41

Example: Psychology

Understand how emotions shape social decisions

Celso de Melo, Peter Carnevale, Stephen Read and Jonathan Gratch. Reading people’s minds from emotion expressions in interdependent decision making. Journal of

Personality and Social Psychology, 106(1), 2014, pp. 73-88


42

Example: Marketing


43

Example: Marketing


CSCI 534 vs. CSCI 590

CS switching to standard of 4 units per course

Officially CSCI 534 T/Th 2:00 – 3:20

CSCI 590 T/Th 3:20 – 3:50

SyllabusSyllabus

CSCI534-Syllabus.pdf

CSCI534-Syllabus.pdf


ResourcesResources

Tools-and-projects.pdf



General Toolkits

Virtual Human Toolkit: contains a number of sensing, language

and synthesis tools. Allows one to construct interactive

affectively-aware digital characters

Litebody another a set of tools for building interactive characters.

ResourcesResources

https://vhtoolkit.ict.usc.edu/

http://relationalagents.com/litebody.html




General Toolkits

Virtual Human Toolkit: contains a number of sensing, language

and synthesis tools. Allows one to construct interactive

affectively-aware digital characters

ResourcesResources

https://vhtoolkit.ict.usc.edu/




Affective Sensing

Expression recognition

FACET – a high-quality commercial facial action unit recognizer

from Emotient. Formally available via research license as CERT.

ResourcesResources

http://mplab.ucsd.edu/~marni/Projects/CERT.htm




Affect generation

SmartBody – character animation system

NVBG – Nonverbal Behavior Generation System

ResourcesResources




Data bases and Lexicons

World Cup Twitter Sentiment dataset: Tweets and twitter

sentiment from 2014 World Cup

ResourcesResources




Projects and Applications

Cognitive

Physical

Social

Central questions:

– what are the cognitive functions of emotion?

– how can they be modeled computationally?

– Can these functions be abstracted and inform AI

– what is the link between cognition and behavior?

– how is emotion physically manifest?

– Can we infer emotional state from behavior?

– how do emotional displays impact the cognition and behavior of users?

– Can synthetic emotions similarly impact users



CognitiveAntos & Pfeffer. "Using emotions to enhance

decision-making.“ AAAI 2011We present a novel methodology for decision-making by computer

agents that leverages a computational concept of emotions. It is

believed that emotions help living organisms perform well in

complex environments. Can we use them to improve the decision-

making performance of computer agents? We explore this

possibility by formulating emotions as mathematical operators that

serve to update the relative priorities of the agent's goals….



Cognitive

Explaining Invasive Ant Scenarios Using

Emotion Modeling

Joshua Wainer, Dusan Jan, and MinHee Kwon

Native ant species play important roles in the stability of their

ecosystems. When native ant species are displaced or killed off by an

invasive ant species, ecosystems that were once stable are usually made

unstable by the arrival of an invasive ant species, however the exact

mechanisms through which such upheaval takes place are rarely

understood. We will attempt to use a simple computational model of

emotions to synthesize both the local interactions among invasive and

native ant species as well as the overall displacement of the latter by

the former in an attempt to give insight into this process



Physical



Physical

Emotion recognition using neutral acoustic speech models

Emily Mower, Takayuki Hidehira and Carlos Busso

Since emotional speech can be regarded as a variation of neutral (non-

emotional) speech, it is expected that a robust neutral speech model can be

useful in contrasting different emotions expressed in speech. The study

presented explores this idea by creating acoustic models trained with

spectral features, measured from the emotionally-neutral TIMIT corpus. ….

This method obtained accuracy levels of up to 78% and 65%, respectively,

can be achieved in the binary and category emotion discriminations. …

These results suggest that well-trained neutral acoustic models can be

effectively used as a front-end for emotion recognition, and once trained

with MFB, it may work well regardless of the channel characteristics.



Social

Rapport (Tickel-Degnen and Rosenthal 1990)

– Positive emotions (e.g. smiles or head nods)*

– Mutual attentiveness (e.g. mutual gaze)

– Coordination (e.g. synchronized movements)

Correlates with socially desirable outcomes:– Liking, trust (Chartrand 1999, Lakin 2003)

– Engagement, willingness to communicate (Tatar 1997; Smith 2000)

– Conversational fluency (Kraut, Lewis et al. 1982; Bavelas, et al. 2000)

– Success in negotiations (Drolet & Morris, 2000)

– Improved test performance in classrooms (Fuchs, 1987)

– Improved quality of child care (Burns, 1984)

– Effective clinical interactions (Philippot, et al., 2003)



Social

The Emotion Mirror: Recognizing Emotion in

Speech and Displaying the Emotion in an Avatar

Abe Kazemzadeh, Samuel Kim, and Yoonji Kim

This paper describes the Emotion Mirror, a demonstration that recognizes emotion in a user's utterance and displays the emotion through the facial expressions of an animated avatar that repeats what the user has said. The demonstration is discussed in terms of system architecture and component design. In addition to the technical details of the system we also look at general engineering and theoretical issues that pertain to the demo system. Finally, describe our experience in creating the system to highlight what worked and what did not.


Summary of Course Goals

Develop a basic understanding of emotion theory

and findings from social sciences

Understand functions of emotions in context of

autonomous systems and human-machine

interaction

Understand state-of-art in affective computing

techniques

Understand methodological issues in collecting and

analyzing data on human emotions

Develop hands-on experience in developing an

affective-computing application/study

CSCI 534 Affective Computing

Documents

Transcript of CSCI 534 Affective Computing