Face2Face: Real-time Face Capture and...

IEEE 2016 Conference on

Computer Vision and Pattern

Recognition

Face2Face:

Real-time Face Capture and Reenactment of

RGB-Videos

Justus Thies1, Michael Zollhöfer2, Marc Stamminger1,

Christian Theobalt2, Matthias Nießner3

1University of Erlangen-Nuremberg

2Max-Planck-Institute for Informatics

3Stanford University



Recognition

Related Work

• Offline • Online

RG

B-D

Real-time Expression Transfer for Facial Reenactment

Vdub: Modifying Face Video of Actors forPlausible Visual Alignment to a Dubbed Audio Track

Creating a Photoreal Digital Actor:The Digital Emily Project

RG

B

Face2Face: Real-time Face Capture and ReenactmentOf RGB-Videos

Spe

cial

Har

dw

are



Recognition

ResultsReenactmentFace CaptureFace Model

Overview

• Parametric Face Model



Recognition


Overview


• Face Capture• Energy Formulation

• Non-rigid Model-based Bundling



Recognition


Overview




• Reenactment• Mouth Retrieval

• Comparisons



Recognition


Overview




• Reenactment• Mouth Retrieval

• Comparisons

• Results / Live Demo



Recognition

Parametric Face Model



Recognition



𝑷



Recognition



𝑷 = 6

𝑷 =

Φ𝛼𝛽𝛿𝛾



Recognition



𝑷 = 6𝑷 = 6+80

𝑷 =

Φ𝛼𝛽𝛿𝛾



Recognition



𝑷 = 6+80𝑷 = 6+80+80

𝑷 =

Φ𝛼𝛽𝛿𝛾



Recognition



𝑷 = 6+80+80𝑷 = 6+80+80+76

𝑷 =

Φ𝛼𝛽𝛿𝛾



Recognition



𝑷 =

Φ𝛼𝛽𝛿𝛾

𝑷 = 6+80+80+76𝑷 = 6+80+80+76+27=269



Recognition



𝑃



Recognition

Face Capture



Recognition


Energy Formulation

𝐸 𝑃 =



Recognition


Energy Formulation

Distance inRGB Color Space

ColorConsistency

𝐸 𝑃 = 𝐸𝑐𝑜𝑙 𝑃

𝒍𝟐,𝟏 − 𝒏𝒐𝒓𝒎



Recognition


Energy Formulation

Distance inImage Space

ColorConsistency

FeatureSimilarity

𝐸 𝑃 = 𝐸𝑐𝑜𝑙 𝑃 +𝐸𝑚𝑟𝑘 𝑃



Recognition


Energy Formulation

RegularizationColorConsistency

FeatureSimilarity

𝐸 𝑃 = 𝐸𝑐𝑜𝑙 𝑃 +𝐸𝑚𝑟𝑘 𝑃 +𝐸𝑟𝑒𝑔(𝑃)

−𝟑 𝝈 +𝟑 𝝈𝟗𝟗, 𝟕%



Recognition


Non-rigid Model-based Bundling

𝐸𝑡𝑜𝑡𝑎𝑙 𝑷 =

𝑖=0

𝑛

𝐸𝑖 𝑷 → 𝑚𝑖𝑛



Recognition



• Iterative Reweighted Least Squares (IRLS)

Gauss-Newton: 𝑱𝑻𝑱𝚫𝑷 = −𝑱𝑻𝑭

𝑱(𝑷) =



Recognition



Inp

ut

Mo

de

l

Hierarchy Levels



Recognition


Tracking



Recognition

Tracking Comparison



Recognition


Tracking Comparison



Recognition

Reenactment



Recognition


ReenactmentOnline RGB-Tracking

Preprocessed Video Tracking

Identity

Expression

Illumination

PoseP

er

Fram

e

Identity

Expression

Illumination

Pose

Pe

r Fr

ame

Reenactment

Expression Transfer

Mouth Retrieval

Compositing

Sou

rce

Act

or

Targ

et A

cto

r



Recognition


Mouth-Retrieval



Recognition


Reenactment Comparison



Recognition

Live-Demo



Recognition


Limitations / Future Work

• Assumption of Lambertian surface and smooth illumination

• No occlusion handling

• No person specific details (fine scale details / wrinkles)

• Reenactment relies on a training sequence (Mouth retrieval)



Recognition


Conclusion

• First Real-time Facial Reenactment only based on RGB-videos• Non-Rigid Model-Based Bundling

• Sub-Space Deformation Transfer

• Image-Based Mouth Synthesis



Recognition

Thank You!



Recognition


References• O. Alexander, M. Rogers, W. Lambeth, M. Chiang, and P. Debevec.

The Digital Emily Project: photoreal facial modeling and animation.In ACM SIGGRAPH Courses, pages 12:1–12:15. ACM, 2009.

• P. Garrido, L. Valgaerts, H. Sarmadi, I. Steiner, K. Varanasi, P. Perez, and C. Theobalt.Vdub: Modifying face video of actors for plausible visual alignment to a dubbed audio track.In Computer Graphics Forum. Wiley-Blackwell, 2015.

• F. Shi, H.-T. Wu, X. Tong, and J. Chai.Automatic acquisition of high-fidelity facial performances using monocular videos.ACM TOG, 33(6):222, 2014.

• C. Cao, Y. Weng, S. Zhou, Y. Tong, and K. Zhou.Facewarehouse: A 3D facial expression database for visual computing. IEEE TVCG, 20(3):413–425, 2014.

• J. Thies, M. Zollhöfer, M. Nießner, L. Valgaerts, M. Stamminger, and C. Theobalt.Real-time expression transfer for facial reenactment.ACM Transactions on Graphics (TOG),34(6), 2015.

• V. Blanz and T. Vetter.A morphable model for the synthesis of 3d faces.In Proc. SIGGRAPH, pages 187–194. ACM Press/Addison-Wesley Publishing Co., 1999.

Face2Face: Real-time Face Capture and...

Documents

Transcript of Face2Face: Real-time Face Capture and...