Towards Accurately Extracting Facial Expression Parameters

Xuecheng Liu Dinghuang Ji Zhaoqi Wang and Shihong Xia{liuxuecheng,jidinghuang,zqwang,xsh}@ict.ac.cn

Institute of Computing Technology, Chinese Academy of Sciences, China

Outline

Background Problem Our Method Experiments Summary

Background

Applications of facial animation

Movies Games Tele-Immersion

Background

• Blendshapes

1

0.8w

2

0.5w

...

Background

Vicon Captured Data

Head Motion

Rotation Transition

Facial Expression

Blendshapes Weights

ProblemUsually,

• Head absolute orientation is first extracted, including head rotation matrix R and transition vector t.

• The captured facial expression data are then transformed to local face coordinates system.

• However, the assumption that four markers are relative fixed may not always be true. This results to error of computing head orientation. Then, above error will be amplified during computing expressions parameters w.

Problem

• [HAV 06] propose a hierarchical manner– Doing a global or gross stabilization choosing

selective markers (head, ears and the nose bone)– Refining the output by local or fine stabilization by

analyzing the marker movements relative to a facial surface model

Our Method

• To undermine the error accumulation, we propose a revision based optimization method.

Our Method

• To undermine the error agglomeration, we propose a revision based optimization method.Simplify

Our Method

• To undermine the error agglomeration, we propose a revision based optimization method.

• Then we use a traditional alternative method and non-negative least square to solve the problem.

Our Method

• Linear blendshape

• Nonlinear blendshape

Experiments

• Data and Environment– Training data： 500 frames– Testing data： 15 segments（ #Frame

>36,000）– Ratio >= 80– Basic blendshapes： 27个– Hardware ： x86 2.67G Core2 CPU， 2G RAM– Software：Matlab

Experiments

• We compare with two published which are both representative works in areas.– [HAV 06] is a typical linear blendshape based

method, this work is presented as course of Siggraph 2006 and used in many popular movies.

– [LXFW 11] is a typical nonlinear blendshape based method, and is published in CGF 2011.

Experiments

• Contrast experiment 1 (with [HAVALDAR 06])[HAVALDAR 06] [Our Method]

Experiments

• Contrast experiment 1 (with [HAV 06])

0%2%4%6%8%

10%12%14%16%18%20%22%

Faci al ani mat i on cl i p

Rela

tive

Err

or

[Hav06] Our Method

0. 00. 20. 40. 60. 81. 01. 21. 4

Faci al ani mat i on cl i p

Abso

lute

Err

or(m

m)

[Hav06] Our Method

Absolute Error

Relative Error

(b)(a)

Experiments

• Contrast experiment 1 (with [HAV 06])

0

5

10

15

Faci al ani mat i on cl i p

Aver

age

Iter

atio

ns

[Hav06] Our Method

020

4060

80100

120

Faci al ani mat i on cl i p

Fram

e ra

tes

f/s

（）

[Hav06] Our Method

(b)(a)

Experiments

Experiments

• Contrast experiment 2 (with [LXFW11])[LXFW11] Our Method]

Experiments

• Contrast experiment 2 (with [LXFW11])Absolute Error

Relative Error

(b)(a)

0. 0

0. 5

1. 0

1. 5

2. 0

2. 5

Faci al ani mat i on cl i p

Abso

lute

err

ormm

（）

[LXFW11] Our Method

0%

5%

10%

15%

20%

25%

30%

35%

Faci al ani mat i on cl i p

Rela

tive

err

or

[LXFW11] Our Method

Experiments

• Contrast experiment 2 (with [LXFW11])

(b)(a)

0

5

10

15

20

Faci al ani mat i on cl i p

Aver

age

iter

atio

ns

[LXFW11] Our Method

0. 0

10. 0

20. 0

30. 0

40. 0

50. 0

Faci al ani mat i on cl i pFr

ame

rate

sf/

s（

）

[LXFW11] Our Method

Experiments

Conclusion

Application Methods Precision promotion

Efficiency promotion

Linear blendshape

Our Method VS. [Hav06]

+12.3% +86.0%

Nonlinear blendshape

Our Method VS. [LXFW11]

+23.5% -18.2%

Bibliography

• [HAV 06] HAVALDAR P. Sony pictures imageworks. In SIGGRAPH ’06: ACM SIGGRAPH 2006 Courses (New York, NY, USA, 2006), ACM, p. 5.

• [LXFW11] Xuecheng Liu, Shihong Xia, Shihong Xia, Yiwen Fan, Zhaoqi Wang. Exploring nonlinear relationship of blendshape facial animation. To appear in the Journal of Computer Graphics Forum.