Automated Prediction of

Preferences Using Facial

Expressions

David Masip, Michael S. North, Alexander Todorov, Daniel Osherson, PLOSOne

1

Automated Prediction of Preferences Using Facial Expressions

Introduction: Reading minds

Detailed problem: subtle facial expressions

• State-of-the-art

• Example

Methods

• Target phase

• Perceiver phase

• Algorithm

Results

• Hypothesis one: simple classifier

• Hypothesis two: machine learning classifier

Conclusions

Índex

2

Automated Prediction of Preferences Using Facial Expressions

1. Reading minds

Can we read the mind? The answer is NO. Not completely at least…

Our face reflects somehow our feelings, emotions, … We propose to read/predict

what people thinks about two stimuli in a very particular setting: Preferences

We explore whether computers can read and infer what humans believe in a binary

choosing scenario, and we use only subtle facial expressions and a laptop.

3

Automated Prediction of Preferences Using Facial Expressions

4

Facial expression is a powerful non verbal communication cue beyond natural language.

“The expression of the Emotions in Man and Animals”. Charles Darwin

Paul Ekman. Universality of Facial expression. 6 Universal human emotions.

“We need to give computers the capacity to read our feelings and react, in ways that

have come to seem startlingly human.” Rana El Kaliouby

1.1. Detailed problem

Automated Prediction of Preferences Using Facial Expressions

5

FACS 46 atomic movements of facial muscles.

Action Units = Building blocks of the emotions

Facial emotions = combinations of AUs

1.1. Detailed problem

Applications: We transmit more data with our expressions than with what we say.

Facial expressions predict: the result of a negotiation, the winner of a congress elections,

the election of a partner,…

GOAL: apply Computer Vision algorithms to model user preferences

in non-posed scenarios.

Automated Prediction of Preferences Using Facial Expressions

6

Which cartoon this student enjoyed the most?

Automated Prediction of Preferences Using Facial Expressions

Methods: target phase

Target phase: Students evaluate two visual stimuli. [People,

Cartoons, Animals, Paintings]

Visualization for 3 seconds in a screen (E-Prime SW)

7

Automated Prediction of Preferences Using Facial Expressions

Methods: perceiver phase

Students infer the preferences of other students from covertly

recorded videos of target’s faces.

They only see the target’s face, and guess the preference

8

Automated Prediction of Preferences Using Facial Expressions

Methods: Algorithm

Facial landmarks detection per frame

Dispersion descriptor based on temporal landmarks displacements

Model:

9

Face Detection

Mesh Fitting

Dynamic Landmark Descriptor

Classifier

Automated Prediction of Preferences Using Facial Expressions

10

Results

Hypothesis one: Maximum – minimum displacement

JESP is human accuracy

Automated Prediction of Preferences Using Facial Expressions

11

Results

Hypothesis two: apply SVM to the 66 differences between the

maximum and minimum centered displacement.

SVM: Standard Machine Learning classifier on a 132 dimensional space

Automated Prediction of Preferences Using Facial Expressions

Conclusions

Complex subtle emotion reading technology

Can we read minds from facial expression?

Yes, if we lower the expectations [Binary preference selection problem]

Applications:

- Depression monitorization

- Pain detection

- Deception detection

- Dynamically price advertising depending on how people responded to it,

- Autism

- Sony, Yahoo!, Motorola, Verizon, …

“Research in this area provides a rare point of convergence

between Computer Science and Social Psychology”

12

Automated Prediction of Preferences Using Facial Expressions