Siggraph Course Mesh Parameterization: Theory and Practice Differential Geometry Primer.
SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and...
-
Upload
theodora-booker -
Category
Documents
-
view
218 -
download
1
Transcript of SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and...
![Page 1: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/1.jpg)
SIGGRAPH Course 30:Performance-Driven Facial AnimationSIGGRAPH Course 30:Performance-Driven Facial Animation
For Latest Version of Bregler’s Slides and Notes please go to:
http://cs.nyu.edu/~bregler/sig-course-06-face/
![Page 2: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/2.jpg)
SIGGRAPH Course 30:Performance-Driven Facial AnimationSIGGRAPH Course 30:Performance-Driven Facial Animation
Section:
Markerless Face Capture and Automatic Model Construction
Part 1: Chris Bregler, NYU
![Page 3: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/3.jpg)
Markerless Face CaptureMarkerless Face Capture
![Page 4: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/4.jpg)
Markerless Face Capture - Overview -Markerless Face Capture - Overview -
• Single / Multi Camera Input
• 2D / 3D Output
• Real-time / Off-line
• Interactive-Refinement / Face Dependent / Independent
• Make-up / Natural
• Flow / Contour / Texture / Local / Global Features
• Hand Crafted / Data Driven
• Linear / Nonlinear Models / Tracking
![Page 5: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/5.jpg)
Markerless Face Capture – History –Markerless Face Capture – History –
Kass, M., Witkin, A., & Terzopoulos, D. (1987) Snakes: Active contour models.
• Single Camera Input
• 2D Output
• Off-line
• Interactive-Refinement
• Make-up
• Contour / Local Features
• Hand Crafted
• Linear Models / Tracking
![Page 6: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/6.jpg)
Tracking = Error MinimizationTracking = Error Minimization
Err(u,v) = || I(x,y) – J(x+u, y+v) ||
![Page 7: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/7.jpg)
Tracking = Error MinimizationTracking = Error Minimization
In general: ambiguous using local features
![Page 8: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/8.jpg)
Tracking = Error MinimizationTracking = Error Minimization
Kass, M., Witkin, A., & Terzopoulos, D. (1987) Snakes: Active contour models.
![Page 9: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/9.jpg)
Tracking = Error MinimizationTracking = Error Minimization
Error = Feature Error + Model Error
![Page 10: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/10.jpg)
Tracking = Error MinimizationTracking = Error Minimization
Error = Optical Flow + Model Error
![Page 11: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/11.jpg)
-
Optical Flow (Lucas-Kanade)
Intensity
x
v ?i
I(x ) - J(x + v )i i i
2
I (x ) - I(x ) vit i
2
ilinearize
I J
E(V)
![Page 12: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/12.jpg)
V
-
= E(V)
V
Model
I (1) - I(1) v t 1
I (2) - I(2) v t 2
I (n) - I(n) vt n
...
2
Optical Flow + Model
![Page 13: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/13.jpg)
V
-
= E(V)
V
Model
I (1) - I(1) v t 1
I (2) - I(2) v t 2
I (n) - I(n) vt n
...
2
V = M( )
Optical Flow + Model
![Page 14: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/14.jpg)
V
-
V
Model
Optical Flow + linearized Model
V = M 2
Z + H V
2Z + C
![Page 15: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/15.jpg)
Optical Flow + Hand-Crafted Model
DeCarlo, Metaxas, 1999 Williams et a,l 2002
![Page 16: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/16.jpg)
Optical Flow and PCAOptical Flow and PCA
Eigen Tracking (Black and Jepson)
![Page 17: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/17.jpg)
PCA over 2D texture and contoursPCA over 2D texture and contours
Active Appearance Models (AAM): (Cootes et al)
![Page 18: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/18.jpg)
PCA over 2D texture and contoursPCA over 2D texture and contours
![Page 19: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/19.jpg)
PCA over texture and 3D shapePCA over texture and 3D shape
3D Morphable Models (Blanz+Vetter 99)
![Page 20: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/20.jpg)
Affine Flow and PCA Affine Flow and PCA
![Page 21: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/21.jpg)
3D Model Acquisition3D Model Acquisition
- Multi-view input: Pighin et al 98
![Page 22: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/22.jpg)
Solution for Rigid 3D AcquisitionSolution for Rigid 3D Acquisition
Structure from Motion:
- Tomasi-Kanade-92
Factorization
3D Pose3D rigid Object
![Page 23: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/23.jpg)
Acquisition without prior model ?Acquisition without prior model ?
• No Model available ?
• Model too generic/specific ?
• Stock-Footage only in 2D ?
![Page 24: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/24.jpg)
Solution based on Factorization Solution based on Factorization
- We want 3 things:- 3D non-rigid shape model- for each frame:
- 3D Pose- non-rigid configuration (deformation)
-> Tomasi-Kanade-92:
W = P S
Rank 3
![Page 25: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/25.jpg)
Solution based on Factorization Solution based on Factorization
- We want 3 things:- 3D non-rigid shape model- for each frame:
- 3D Pose- non-rigid configuration (deformation)
-> PCA-based representations:
W = P non-rigid S
Rank K
![Page 26: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/26.jpg)
3D Shape Model 3D Shape Model
Linear Interpolation between 3D Key-Shapes:
S1 S2S
![Page 27: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/27.jpg)
Basis Shape FactorizationBasis Shape Factorization
Complete 2D Tracks or Flow Matrix-Rank <= 3*K
![Page 28: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/28.jpg)
Nonrigid 3D Kinematics from point tracks Nonrigid 3D Kinematics from point tracks
-
![Page 29: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/29.jpg)
-
Nonrigid 3D Kinematics from dense flow
![Page 30: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/30.jpg)
-
Nonrigid 3D Kinematics from dense flow
![Page 31: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/31.jpg)
-
Nonrigid 3D Kinematics from dense flow
![Page 32: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/32.jpg)
Motion Capture
Modeling
Synthesis
Nonrigid 3D Kinematics from dense flow Nonrigid 3D Kinematics from dense flow
![Page 33: SIGGRAPH Course 30: Performance-Driven Facial Animation For Latest Version of Bregler’s Slides and Notes please go to: bregler/sig-course-06-face](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d1c5503460f949f18bc/html5/thumbnails/33.jpg)
Markerless Face Capture - Summary -Markerless Face Capture - Summary -
• Single / Multi Camera Input
• 2D / 3D Output
• Real-time / Off-line
• Interactive-Refinement / Face Dependent / Independent
• Make-up / Natural
• Flow / Contour / Texture / Local / Global Features
• Hand Crafted / Data Driven
• Linear / Nonlinear Models / Tracking