PowerPoint Presentation

CVPR 2016 Tutorial Fitting Surface Models to Data2016/10/02@sumisumith

1

2

http://awf.fitzgibbon.ie/cvpr16_tutorial

2

Fitting Surface Models to Data3CVPR 2016 Tutorial

Andrew Fitzgibbon, MicrosoftJonathan Taylor, PerceptiveIO

3

PeopleFinding Nemo: Deformable Object Class Modelling using Curve MatchingCVPR 10Mukta Prasad, Andrew Fitzgibbon, Andrew Zisserman, Luc Van GoolKintre: Animating the World with the Human BodyUIST 12Jiawen (Kevin) Chen, Shahram Izadi, FitzgibbonThe Vitruvian Manifold: Inferring dense correspondences for one-shot human pose estimationCVPR 12Jonathan Taylor, Jamie Shotton, Toby Sharp, FitzgibbonWhat shape are dolphins? Building 3D morphable models from 2D imagesPAMI 13Tom Cashman, FitzgibbonUser-Specific Hand Modeling from Monocular Depth SequencesCVPR 14Taylor, Richard Stebbing, Varun Ramakrishna, Cem Keskin, Shotton, Izadi, Fitzgibbon, Aaron Hertzmann Real-Time Non-Rigid Reconstruction Using an RGB-D CameraSIGGRAPH 14Michael Zollhfer, Matthias Niener, Izadi, Christoph Rhemann, Christopher Zach, Matthew Fisher, Chenglei Wu, Fitzgibbon, Charles Loop, Christian Theobalt, Marc StammingerLearning an Efficient Model of Hand Shape Variation from Depth ImagesCVPR 15Sameh Khamis, Taylor, Shotton, Keskin, Izadi, Fitzgibbon Efficient and Precise Interactive Hand Tracking through Joint, Continuous Optimization of PoseSIGGRAPH 16and CorrespondencesTaylor, Lucas Bordeaux, Cashman, Bob Corish, Keskin, Sharp, Eduardo Soto, David Sweeney, Julien Valentin, Ben Luff, Arran Topalian, Erroll Wood, Khamis, Kohli, Izadi, Richard Banks, Fitzgibbon, Shotton.Fits Like a Glove: Rapid and Reliable Hand Shape Personalization. CVPR 16David Joseph Tan, Cashman, Taylor, Fitzgibbon, Daniel Tarlow, Khamis, Izadi, Shotton.

4

Learn how to SOLVE HARD VISION PROBLEMS, using tools that may appear inelegant, but are much smarter than they look.Goal5

5

ApplicationsCurve/surface fitting

Parameter estimationBundle adjustment(Video from our friends at Google)

Kintre7

KinEtre: Animating the World with the Human Body Chen et al. UIST 2012

Kintre8

KinEtre: Animating the World with the Human Body Chen et al. UIST 2012

Kintre9

KinEtre: Animating the World with the Human Body Chen et al. UIST 2012

Vitruvian Manifold, CVPR 12

Fitting subdivision surfaces to 2D data

Fitting subdivision surfaces to 2D data

15

Fitting polygon meshes to video

17

[3D Scanning Deformable Objects with a Single RGBD Sensor, Dou et al, CVPR15]Input Kinect StreamKinectFusionDeformable Fusion

17

Hand Tracking

Hand Shape Personalization:CVPR 2014, CVPR 2015, CVPR 2016Discriminative Hand Pose ReinitializationICCV 2015, CHI 2015

Hand Pose Estimation via Model Fitting (read Hand Tracking)CHI 2015, SIGGRAPH 2016

Image Denoising19[STRANDMARK & AGARWAL, 2014, arXiv:1403.5590]

Matrix Factorization [Hong & F., ICCV 15]

Matrix Factorization [Hong & F., ICCV 15]

Myth: You dont need to optimize far22

For each task, the method is the same23

So, you can do lots of things by fitting models to data.

How do you do it right?

Lets look at some examples.

24

Continuous Optimization

Andrew FitzgibbonMicrosoft Research Cambridge

Goal

Classes of functionsquadraticconvexquasiconvexmulti-extremumnoisyhorrible

27

Classes of functionsquadraticconvexquasiconvexmulti-extremumnoisyhorrible

28

29quadraticconvexquasiconvexmulti-extremum

29

30quadraticconvexquasiconvexmulti-extremumEasyHard

30

Derivatives31

01332 440 955 o2 sian31

Alternation

EasyHard

33

Rosenbrock

33

Gradient DescentAlternation is slow because valleys may not be axis alignedSo try gradient descent?

-5

0

5

10

15

-5

0

5

10

15

Gradient DescentAlternation is slow because valleys may not be axis alignedSo try gradient descent?

Gradient DescentAlternation is slow because valleys may not be axis alignedSo try gradient descent?

Note that convergence proofs are available for both of the aboveBut so what?

And on a hard problem

Use a better algorithm(Nonlinear) conjugate gradientsUses 1st derivatives onlyAvoids undoing previous work

Use a better algorithm(Nonlinear) conjugate gradientsUses 1st derivatives onlyAnd avoids undoing previous work101 iterations on this problem

but we can do better

Use second derivatives

Use second derivatives

Use second derivativesHow does it look?

43

Use second derivatives

Use second derivatives

Use second derivatives

Use second derivatives

1st derivatives again

Back to first derivatives

50

Back to first derivatives

Order N Cubed?

Typical Hessian structure

53

54 http://bit.ly/2dcaFr7

Conclusion: YMMV

Conclusion: YMMVGIRAFFE

500 runs

Conclusion: YMMV

FACE

1000 runs

DINOSAUR

1000 runsGIRAFFE

500 runs

On many problems, alternation is just fineIndeed always start with a couple of alternation stepsComputing 2nd derivatives is a painBut you dont need to for LM

There is no universal optimizer

What is a surface?

59

00.20.40.60.81

1.922.12.22.32.4

Curvesfunction y(x::Interval)::Real = .3*x + 2

function C(t::Interval)::Point2D = Point2D(t^2 + 2, t^2 t + 1)

function S(u::Interval, v::Real)::Point3D = Point3D(cos(u), sin(u), v)

60

22.22.42.62.83

0.750.80.850.90.951

Surface61

61

Surface62

62

Surface63

63

TOOL: Subdivision Surfaces 64

And I still havent told you about the surface model

64

Control mesh65

Limit Surface66

Subdiv rule: Step 1. Add new vertices67

Subdiv rule: Step 2. Average neighbours68

2 subdivisions69

3 subdivisions70

Limit Surface71

Control vertices define the shape72

Subdivision Surface: Parametric form73

73

examples74

Back to Dolphins

75

What Shape are Dolphins? Building 3D Morphable Models from 2D Images

76

77Loop Subdivision Surface 3DBB3D

Non-rigid 3D B

3D

( Non-rigid )

77

Model representation

78

79

80

81

Data terms

82

Data terms83Camera positionProjectione.g. Perspective

Data terms84

Gaussian shape weights

Smooth contour

85

86

Continous optimization

87

Initial estimate for mean shapeThis is true, but misleading

88

Initial estimate for mean shape

True initial estimate: only the topology is really important.But the easiest way to get the topology is to build a rough template.89

Initial estimate for mean shape

True initial estimate: only the topology is really important.But the easiest way to get the topology is to build a rough template.90

91

Example results92

Example results93

Optimization

94

Number of images95

81632

Parameter sensitivityPixel terms: noise level paramsDimensionless terms

Smoothness terms

96

96

97

98

99

100

101

102Fitting Surface Models to Data

(^^)

@sumisumith

103