Affective Computing: Machines with Emotional Intelligence
description
Transcript of Affective Computing: Machines with Emotional Intelligence
Affective Computing: Machines with Emotional Intelligence
Hyung-il Ahn
MIT Media Laboratory
• Expressing emotions
• Recognizing emotions
• Handling another’s emotions
• Regulating emotions \
• Utilizing emotions /
(Salovey and Mayer 90, Goleman 95)
Skills of Emotional Intelligence:
if “have emotion”
We have pioneered new technologies to recognize human affective information:
Sensors, pattern recognition and common sense reasoning to infer emotion from physiology, voice, face, posture & movement, mouse pressure
Mind-Read: Recognizing complex cognitive-affective states from joint face and head movements
Future “teacher for every learner”
Can we teach a chair to recognize behaviors indicative of interest and boredom? (Mota and Picard)
Sit upright Lean Forward Slump Back Side Lean
What can the sensor chair contribute toward inferring the student’s state: Bored vs. interested?
Results (on children not in training data, Mota and Picard, 2003):
9-state Posture Recognition: 89-97% accurateHigh Interest, Low interest, Taking a Break: 69-83% accurate
Detecting, tracking, and recognizing facial expressions from video (IBM BlueEyes camerawith MIT algorithms)
Complex Mental States(subset)
Concentrating
Disagreeing
Interested
Thinking
Unsure
AbsorbedConcentratingVigilant
DisapprovingDiscouragingDisinclined
AskingCuriousImpressedInterested Brooding
ChoosingThinkingThoughtful
BaffledConfusedUndecidedUnsure
Affective-Cognitive Mental StatesBaron-Cohen et al. AUTISM RESEARCH CENTRE, CAMBRIDGE
Agreeing
AssertiveCommittedPersuadedSure
Knows when a person/customer is:• Concentrating, and does not interrupt unless very
important• Thinking, and can pause to let you think• Unsure, and can offer to explain differently• (Not) interested in what it says• (Dis)agreeing, and can adjust response respectfully
Technology that understands and responds to human experience like a caring, respectful person would, for example:
Technology with people sense will perceive cognitive-affective states, e.g., before interrupting
hmm … Roz looks busy. Its probably not a good time to bring this up
Analysis of nonverbal cues
Inference and reasoning about mental states
Modify one’s actionsPersuade others
Feature point tracking
Head pose estimation
Facial feature extraction
Head & facial action unit recognition
Head & facial display recognition
Mental state inference
Hmm … Let me think about this
Experimental Evaluation ConclusionsInferring Cognitive-Affective State from Facial+Head movements (el Kaliouby, 2005)
Other examples:
Agree
Disagree
75% sit in front of computers (static)
Back pain/injury = #2 cause of missed work
Physical movement helps prevent/reduce back pain
Goals :- Fostering healthy posture- Building social rapport- Improved task performance
(Affect-Congruent behavior)
Robotic Computer (RoCo) :World’s first physically animated computer
Animated Desktop Monitor: RoCo = Robotic Computer
QuickTime™ and aCinepak decompressor
are needed to see this picture.
RoCo Behavior
QuickTime™ and aMPEG-4 Video decompressor
are needed to see this picture.
• NOT when: you’re concentrating, interested, in the middle of an engaging task, or otherwise attentive/focused on the monitor’s content.
• Might make a micro-movement when you’re looking away or blinking in the middle of a task.
• Might make a larger movement to attract a new user, bow to welcome, or when user shifts tasks and hasn’t shifted posture (etc.)
When should RoCo move? (Future work & not topic of this paper, but important to mention)
RoCo’s postures congruous to the user affect
N=(17)
“Stoop to Conquer” : Posture and affect interact to influence computer users’ comfort and persistence in problem solving tasks
People tend to be more persistent and feel more comfortable when RoCo’s posture is congruous to their affective state
“Stoop to Conquer”: Posture congruent with emotion improves persistence (# tracing attempts, two different experiments)
RoCo’s Posture:
Human State:
Slumped Neutral Upright
Success (“you scored 8/10”) N=30
8.2 8.3 12.0
Failure (“you scored 3/10”) N=19
9.6 7.4 6.9
A multi-modal affective-cognitive measures for product evaluation with computational models of predicting customer decisions
Predicting customer product preferences by combining information about emotion and cognition
We are creating new computational models to measure human affective experience and to predict human decision-making & preference
Background findings to inform new research:
The brain uses both emotion (affect) and cognition in decision making
-> model should combine both affect and cognition
A person in an experiment is likely to cognitively bias their self-report of what they like.
-> method should not rely on only self-report
When a person is cognitively loaded they are more likely to use emotion in decision-making.
-> method should slightly load person cognitively
Background findings to inform new method:
Multiple measures of affect provide most robust assessment:
-> method can measure affective physiology (face, skin conductance) as well as behavior and self-report
Sweeter beverages are preferred on the first sip; long-term accumulation of something mildly bad is required before it is “bad enough to notice”
-> method should require lots of sips of every beverage
More complete understanding of consumer desire
Skin Conductance
ANTICIPITORY FEELING
Arousal
Multi-Dimensional
Response Physical
NUMBER OF SIPSAmount Consumed
Facial Expression
AFFECTIVE LIKING
Emotions
Self ReportCOGNITIVE LIKING
Purchase intent Liking
Expectation
Videos of Testing• Here is a sneak preview of my project. Make sure to look
for consumers emotions that may not be captured in self reported questions.
QuickTime™ and aYUV420 codec decompressor
are needed to see this picture.
Test Products
• Stronger Performer – – Pepsi Vanilla
• Performed in top 25%, green region, in Directions HUT
• Weaker Performer – – Pepsi Summer mix
• Performed in lower 40%, lower yellow region, in Directions HUT
Products chosen with clear performance differences
• Two techniques performed simultaneously – Facial Imaging and Head Positioning
Tracking face muscle movements to interpret emotions
– Galvanic Skin Response (GSR)Measures Arousal, used as an intensity measure for emotions
Affective Computing
Facial HeadExpression Position
• Concentrating
• Thinking
• Confused
• Interested
• Agreeing• Disagreeing
Affective-Cognitive Mental States
GSR Shows Intensity
+ = Interpretation
Method: Choice Technique
• Choice technique - respondent selected one of two vending machines
– Process is repeated 30 times– Eventually respondents realized each machine favors a
different product and will select the vending machine hoping to receive their favored product
– 70/30 probability of either product coming out of either machine
Two cups on each side of the computer: Pepsi Vanilla and Pepsi Summer Mix
Use of straws avoided blocking facial reaction
Method - General Set-Up
Machine 2Machine 1
135 246 135 246
Experimental Set Up
Machine Selection Sip on Resulted Beverage Answer Questions
Method - Step 1
• Each vending machine directed you to sip a beverageRANDOMLY CHOOSE A VENDING MACHINE
Method- Step 2
RESPONDENTS SIP RESULTED BEVERAGE
Method – Step 3• Answer Questionnaire used in standard CLT
– Overall Liking (beverage and machine)– Purchase Intent, Comparison to Expectation
• Reselect a machine • 30 machine selections were made
Method – Step 4
Data collection timeline
Start
SelectOutcome
Evaluate Start (Next trial)
Choice 2 70% Mix30% Vanilla
Choice 1 70% Vanilla30% Mix
Measuring
ANTICIPITORY FEELING(hope/dread)
Skin conductance
vanillaor mix
Sip
How muchdo you like
the sip?
Measuring AFFECTIVE LIKING
(initial reaction)
Facial expressionSkin conductance
Measuring
COGNITIVE LIKING
(post reaction)
Self-report
Question
Data collected throughout experiment
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
Videos of Testing
QuickTime™ and aYUV420 codec decompressor
are needed to see this picture.
Videos of Testing
QuickTime™ and aYUV420 codec decompressor
are needed to see this picture.
Videos of Testing
QuickTime™ and aYUV420 codec decompressor
are needed to see this picture.
Videos of Testing
QuickTime™ and aYUV420 codec decompressor
are needed to see this picture.
Analysis• Our hypothesis is that joining quantitative and qualitative methodologies will help provide understanding of consumers’ real product evaluations
Discussion
Skin Conductance
ANTICIPITORY FEELING
Arousal
Multi -Dimensional
Response
PhysicalNUMBER OF SIPSAmount Consumed
Facial Expression
AFFECTIVE LIKING
Emotions
Self ReportCOGNITIVE LIKING
Purchase intent Liking
Expectation