Graphite 2004Statistical Synthesis of Facial

Expressions for the Portrayal of Emotion

Lisa GralewskiBristol UniversityUnited Kingdomgralewsk@cs.bris.ac.uk

Aim and Motivation:

Generate new emotion specific video sequences.

Facial expressions + head motion. Potentially infinitely long No repeated footage

Provide an animation tool: User driven. Allows emotion selection. Automatic generation of novel facial expression

sequences.

Approach: Statistical Approach

Divides into 2 stages

Stage 1 Modelling: To form generative emotion models.

• Shape model •point data, potential to drive animations

• Combined appearance space model•texture and shape for the generation of new video frames.

Stage 2 Synthesis: Method to generate new video footage / point data

Emotion video footage is considered as a time series of changing facial expressions.

Time series can be modelled with Auto regressive processes (ARPs) Advantages:

constructed models generate new data similar to training data. Potentially infinitely long generated sequences.

Disadvantages: For successful modelling, input data must have stationary signal.

Use ARP’s to form generative emotion specific models. These allow synthesis of new facial expression emotion sequences.

Stage 1: Modelling Approach

Time

Modelling: Data Collection

Acquire video footage Emotions: Happy, sad

and, angry.

Split video into:

1) Shape representation Using hand labelled

control points.

2) Texture representation Using warped image set

Neutral pose mean shape of control point set.

Modelling: Facial expressions

Apply principle components analysis (PCA) video data to calculate eigenvectors for:

Shape space (point data) Texture space (warped pixel data) Combined space (shape + texture).

Calculate shape and combined space responses.

Data projected from a higher-D space down to lower D-space. Each video frame has a response Together the responses for a sequence form an eigensignature.

Form linear shape and combined space models. Allow reconstruction of original data.

Modelling: Eigensignatures Visualise eigensignature:

Plot of 1st and 2nd principle components (responses).

Each point of the eigensignature represents a single video frame.

AngryHappy

Sad

SHAPE EIGENSIGNATURE

Happy

Sad

Angry

COMBINED EIGENSIGNATURE

Modelling: Construct ARP’s

3 video clips yield: Happy eigensignature Sad eigensignature Angry eigensignature

Consider eigensignature as a multivariate time series.

Separate application of multivariate-ARP’s Results: separate ARP emotion models.

Models used to generate new emotion specific eigensignatures-(stage 2: SYNTHESIS)

Stage 2: Synthesis

Generate eigensignatures Potentially infinitely long Retains look and feel of original.

Original Eigensignature: Emotion Angry

Generated ARPEigensignature: Emotion Angry

Synthesis: Generated Shape ARP sequences:

Original Angry ARP Generated Angry

Using linear shape space PCA models.Project generated shape eigensignature back to shape space.

Synthesis: Generated Shape ARP sequences:

ARP Generated Happy ARP Generated Sad

Synthesis: Generated Combined space ARP sequences:

Original Angry ARP Generated Angry

Using combined appearance PCA models. Project generated eigensignatures back to combined appearance

space.

Synthesis: Generated Combined space ARP sequences:

ARP Generated Happy ARP Generated Sad

Application: User-driven synthesis

Approach: Self organising maps (SOMs) SOM’s project the n-D ARP emotion data into a 2D

image space, defined as ‘expression space’.

Tool development Expression Space visualises emotion ARP models. Navigate space. Automatic generation new emotion specific videos.

Self-Organising Maps

Clustering technique Kohonen Neural Network Visualises n-observations as

g-groups.

SOM splits into 2 elements1) Data – provided by ARP

models Requires sufficient training

data.

2) Weight vectors components. xy position on map After training weight vectors

equivalent to ARP models.

x1,y1

W1, W2 W3 W4

Weight Vectors ARP-coefficients

x2,y2 x3,y3 x4,y4

SOM UNITS

Self-Organising Map Structure

Technique: Acquisition of SOM training data.

COMPLETE EMOTION EIGENSIGNATURE

• Use N emotion ARP’s to train 50x50 unit SOM.

Split complete eigensignature into frame segments with (length-1)

frame overlap.

1 2 N-segments

• Model each segment with an ARP.• Result: obtain N ARP models for SOM training.

Results After training :•Each unit of the SOM is equivalent to a single ARP model.•ARP emotion models are clustered.

Technique: Forming Expression Space

Generate Expression space by colour coding the SOM units. Use an Euclidean distance scheme, based on comparison between the sample ARP

emotion models used for training and SOM map unit ARP’s.

Red units most similar to angry ARP models

Green units most similar

to happy

Blue unit associate to

sad ARP models

Intensity of colour indicates the better match to an emotion group.

Results: SOM Shape Space SOM Combined Space SOM

Emotions cluster into 3 distinct regions. Distinct separation between angry and all other emotions. Large transitional region between sad and happy. Dark regions not closely related to any emotion.

Angry Cluster

Sad Cluster

Happy Cluster

Angry Cluster

Sad Cluster

Happy Cluster

Navigating Expression Space

This tool provides an intuitive interface.

Aimed at non-specialist users

Allows easy traversal of the highly dimensional expression space.

Automatically generates new facial expression sequences for selected emotions.

SOMVisualisation

Generated Expression

Navigating the Combined Appearance Space

SOMVisualisation

GeneratedVideo texture

Complete Eigensignature

Generated Eigensignature

Results: Using different ARP models from the SOM to generate sad crowd.

Mixed Crowd Scene

Conclusions

Demonstrated approach which can reconstruct and generate novel footage.

Successful use of PCA and ARP’s: To construct emotion models that correctly co-

articulate head motion and facial expression.

Eigensignature representation unique for each emotion. Emotion ARP’s cluster distinctly Clustered space easily navigable by non-

specialists.

Finally……

Any questions?