Statistical models for face analysis

4th Stochastic geometry days, August 2015

Pascal Bourdon

XLIM Laboratory (UMR CNRS 7252) / Université de Poitiers

pascal.bourdon@univ-poitiers.fr

2 8/27/2015

Summary

• Introduction

• Early works

• A deformable face model

• Model fitting and data representation

• Conclusion

Statistical models for face analysis

3 8/27/2015

Summary

• Introduction

• Early works

• A deformable face model

• Model fitting and data representation

• Conclusion

Statistical models for face analysis

4

Introduction

8/27/2015

Why face analysis?

5

Introduction

8/27/2015

Why face analysis?

6

Introduction

8/27/2015

Why face analysis?

7

Introduction

8/27/2015

Why face analysis?

• Identity recognition

• Morphing/VFX

• Aging simulation

• Fatigue monitoring (road safety)

• Behavior analysis (reactions, emotions…)

• Virtual mirror (online shopping: makeup, glasses…)

• Visio-conference, videos games…

8

Introduction

8/27/2015

A challenging task…

Pose

Illumination

Expression

Resolution

9 8/27/2015

Summary

• Introduction

• Early works

• A deformable face model

• Model fitting and data representation

• Conclusion

Statistical models for face analysis

Matching a reference image (template)

Object tracking associates image objects in consecutive video frames

10

Early works: object tracking/image alignment

8/27/2015

),(,,,',' yxtyxIttyxI

Matching a reference image (template)

Image alignment associates target objects from a reference

11

Early works: object tracking/image alignment

8/27/2015

),(,',' yxyxTyxI

Both problems are traditionally solved using optical flow estimation

• Common assumption: flow is constant within the image object I(x,y)

• Example: least mean squares method (Lucas-Kanade)

12

Early works: object tracking/image alignment

8/27/2015

ttVyyVxxItyxI ,,,,

Vy

Vxυ

υ),,(),,( tyxItyxI T

t

),(

2),,(),,(

yx

T

t tyxItyxI υυ

0ˆ

υ

υ

Lucas-Kanade tracker

13

Early works: object tracking/image alignment

8/27/2015

14

Early works: object tracking/image alignment

8/27/2015

A constant flow within the image object?

15

Early works: object tracking/image alignment

8/27/2015

A constant flow within the image object?

),(

2

,,yx

warptemplate

yxIyxT WW

yyz

xzx

tysxw

tywxsyx,W

Matthews-Baker ICIA tracker

16

Early works: object tracking/image alignment

8/27/2015

17

Early works: object tracking/image alignment

8/27/2015

A constant flow within the image object?

Rigid objects : 3D-to-2D projections

Non-rigid objects : a deformable model is needed!

18 8/27/2015

Summary

• Introduction

• Early works

• A deformable face model

• Model fitting and data representation

• Conclusion

Statistical models for face analysis

From rigid to deformable models

We want to enhance a previously static alignment model (i.e. template image) to

take into account changes of identity, expression or luminosity. In computer

vision, these changes are seen as local variations of shape and/or texture

19

A deformable face model

8/27/2015

Active Appearance Models (Cootes et al.)

are able to jointly synthesize an image (texture) and the shape it relates to. An

AAM model is built through principal component analysis (PCA) of aligned texture

and shape data from a (manually annotated) image database

20

A deformable face model

8/27/2015

21

A deformable face model

8/27/2015

Active Appearance Models (Cootes et al.)

are able to jointly synthesize an image (texture) and the shape it relates to. An

AAM model is built through principal component analysis (PCA) of aligned texture

and shape data from a (manually annotated) image database

22

A deformable face model

8/27/2015

Active Appearance Models (Cootes et al.)

are able to jointly synthesize an image (texture) and the shape it relates to. An

AAM model is built through principal component analysis (PCA) of aligned texture

and shape data from a (manually annotated) image database

23

A deformable face model

8/27/2015

Face synthesis from an AAM model

sP

p

pp

1

0 sss

gP

p

pp

1

0 AAAShape model: Texture model:

24 8/27/2015

Summary

• Introduction

• Early works

• A deformable face model

• Model fitting and data representation

• Conclusion

Statistical models for face analysis

Fitting a model to new data

• Original AAM (Cootes) : generating shape samples from the database

with random noise and learning a regression model which minimizes the

error between observed and synthesized textures

• Gradient-descent method (Matthews-Baker) : a least mean squares-

based approach inspired by Lucas-Kanade, where AAM parameters are

estimated together with global transform parameters

25

Model fitting and data representation

8/27/2015

),(

2

,;,;,,,yx

warptemplate

yxIyxA ΦpWΨΦpΨ

ΦΨ, : AAM parameters p : global (e.g. affine) transform parameters

W: image warping function, computed from all geometry-related parameters

Fitting a model to new data

26

Model fitting and data representation

8/27/2015

27

Model fitting and data representation

8/27/2015

Further data analysis from estimated AAM parameters

Motion capture and animation

28

Model fitting and data representation

8/27/2015

Further data analysis from estimated AAM parameters

Emotion analysis (e.g. online learning)

29 8/27/2015

Summary

• Introduction

• Early works

• A deformable face model

• Model fitting and data representation

• Conclusion

Statistical models for face analysis

30 8/27/2015

Conclusion

• A statistical face model learned from observed data

• Compacts relevant information into just a few parameters

• Applications in the field of information retrieval

• Current work : polynomial representation of texture model

and applications in animation and e-learning

• Next : force semantic information into PCA analysis i.e.

individual muscle activity parameters (local constraints)

Statistical models for face analysis

31 8/27/2015

Thank you !

Applications in alignment, tracking and landmarking