2011/5/23

3D Morphable Model Based Face Replacement in Video

Presenter ： Bo-Hung ChenAdviser ： Dr. Shih-Chung Chen

Yi-Ting Cheng, Virginia Tzeng, Yung-Yu Chuang, Ming OuhyoungDept. of Computer Science and Information Engineering

National Taiwan University(2008)

2011/5/23

Outline• Motivation• Introduction• System structure• 3D Face Database• Expression Model Database• Estimation, Correction and Compared• Results• References

2011/5/23

Motivation

• Entertainment

• To assess Plastic surgery

Why this paper?

2011/5/23

• In the past, The most naive method is to ask the source subject to act the same as the target subject under the similar lighting condition

• In this paper, we present a system for face replacement in video to replace the target subject face in the target video with the source subject face, under similar pose

Introduction

2011/5/23

System structure

2011/5/23

3D Face Database• The morphable model of 3D faces is a vector space of 3D shapes and

textures spanned by a set of example faces

• This morphable model is derived from structural light 3D scans of 117 adults (92 males and 25 females). A correspondence algorithm makes all the faces fully correspondent. Each 3D face model is represented by 65536 vertices with textures.

2011/5/23

Profile line of the model• Furthermore, to match the profile line of the model better, we define an

energy term to match the profile line between the height of eyes and jaw.

• At each horizontal scan line, an energy term restricts the x coordinate of the profile line in Iinput close to the one in Imodel :

• Where xp,i and x’p,i are the x coordinate of the profile line on the i-th horizontal scan line in Iinput and Imodel respectively

Energy term ：能 [ 量 ] 項 ( 物理學專有名詞 )

Norm : 範數，是具有「長度」概念的函數，為向量空間內的所有向量賦予非零的正長度或大小。

2011/5/23

Principal Component Analysis• PCA is used to perform a basis transformation to an orthogonal coordinate s

ystem formed by the eigenvectors eS,i and eT,i in descending order according to the eigenvalues σS,i and σT,i of the covariance matrices

2011/5/23

Expression Database• We have to extend the neutral source face model to expressional model. • 13 key expressions :

– Emotional expressions : angry, smiling, happy, sad, and surprised.– Verbal key expressions : pronouncing ‘a’, ‘e’, ‘uh’, ‘m’, ‘o’, and ‘u’.– Expressions of closing eyes (Left and Right)

• Each model in the expression database is represented by 436 vertices, and we manually match these vertices to the 65536 vertices of the reconstructed model.

2011/5/23

Estimation, Correction and Compared• In this section, we introduce the head pose estimator first, and then we

introduce the lighting estimator and face relighting module in detail.

2011/5/23

Estimation Euclidean Distance• Given an input image, the goal is to minimize the Euclidean distance over all

color channels and all pixels between the input image Iinput and the image

Imodel synthesized from the current model.

• To match the geometry of model better, we exploit the labeled feature points (qx,i , qy,i ) and the image-plane position (px,ki , py,ki) of the

corresponding vertices ki in an additional feature term.

2011/5/23

Pose Estimation• Based on the set of 87 feature points which are detected by the face ali

gnment module and the corresponding preselected feature points in the target face model, we can estimate pose parameters by minimizing the error E between them.

• Where wi is the weight of the i-th feature point, (qx,i , qy,i ) is the position of the i-th feature point of face alignment, and (px,i , py,i ) is the project

position of the i-th feature point of the target model.

2011/5/23

Corrected Overfitting E ects ff• Minimization of these energy functions with respect to α, β, ρ may cause

overfitting e ects. Therefore, we employ a maximum a posteriori estimator ff(MAP). Finally, the posteriori probability is then maximized by minimizing

Posteriori probability ： 後端驗證概率 ( 物理學專有名詞 )

Maximum a posteriori estimator (MAP) ：極端後端驗證

2011/5/23

Face Relighting• If the source face and the target face were under di erent illumination, the rff

eplacement result would appear perceptually unreasonable, so we need to adjust skin color and lighting of the source face

• where ρc is the average color of each color channel, ac,k are the spherical harmonic coe cients which we want to estimate as lighting parameters, H are ffithe spherical harmonics, and n(x, y) is the surface normal at the image location (x, y). We use the 3D face models to render normal maps of both the target face and source face

2011/5/23

Matching the Mouth and Blinking

2011/5/23

Result

2011/5/23

Results

2011/5/23

References[1] The prestige, 2006.[2] R. Basri and D. W. Jacobs. Lambertian reflectanceand linear subspaces. IEEE Transactions on Pat-tern Analysis and Machine Intelligence, 25(2):218–233, 2003.[3] D. Bitouk, N. Kumar, S. Dhillon, P. N. Belhumeur,and S. K. Nayar. Face swapping: automaticallyreplacing faces in photographs. ACM Transactionson Graphics (SIGGRAPH), 27(3), 2008.[4] V. Blanz, C. Basso, T. Poggio, and T. Vetter. Re-animating faces in images and video. ComputerGraphics Forum, 22:641–650, 2003.[5] V. Blanz, K. Scherbaum, T. Vetter, and H.-P. Sei-del. Exchanging faces in images. Computer Graph-ics Forum, 23(3):669–676, 2004.[6] V. Blanz and T. Vetter. A morphable model forthe synthesis of 3d faces. In Computer GraphicsProc. SIGGRAPH’99, pages 187–194, 1999.

2011/5/23

[7] Y. Liang. Image based face replacement in video.Master’s thesis, CSIE Department, National Tai-wan University, 2009.[8] Z. Liu, Z. Zhang, C. Jacobs, and M. Cohen. Rapidmodeling of animated faces from video images. InProceedings of ACM International Conference onMultimedia, pages 475–476, 2000.[9] J. A. Nelder and R. Mead. A simplex method forfunction minimization. Computer Journal, 7:308–313, 1965.[10] P. P´erez, M. Gangnet, and A. Blake. Poisson im-age editing. ACM Transactions on Graphics (SIG-GRAPH), 22:313–318, 2003.[11] F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, andD. H. Salesin. Synthesizing realistic facial expres-sions from photographs. In SIGGRAPH ’06: ACMSIGGRAPH 2006 Courses, page 19, 2006.[12] H. Pyun, Y. Kim, W. Chae, H. W. Kang, andS. Y. Shin. An example-based approach for facialexpression cloning. In 2003 ACM SIGGRAPH /Eurographics Symposium on Computer Animation,pages 167–176, 2003.[13] J. yong Noh and U. Neumann. Expressioncloning. In SIGGRAPH ’06: ACM SIGGRAPH2006 Courses, page 22, 2006.

2011/5/23

Thanks for your attention