1

From 2D to 3DBuilding 3D Morphable Models from 2D Images

Student: Shangling Yang & Weihao Gan

Advisor: Prof. Mahdi Soltanolkotabi

2

Introduction

System Input

Energy Formulation

Optimization Algorithm

Experimental Results

Outline

Cashman, Thomas J., and Andrew W. Fitzgibbon. "What shape are dolphins? Building 3D morphable models from 2D Images." Pattern Analysis and Machine Intelligence, IEEE Transactions on 35.1 (2013): 232-244.

3

1. Introduction3DMM (3D Morphable Model) is an very active topic in current computer vision research. 3DMM statistically builds a joint model based on 3D shape and texture in 2D.

• Single-view reconstruction

• Rigid reconstruction from multiple views

• Nonrigid reconstruction from multiple views

4

1. Introduction

A rigid template triangulation (red lines) which we use to initialize our dolphin model. Note that this triangulation defines only the mesh topology, and is defined just once per object class, not per image. The Loop subdivision

surface defined by combining this triangulation with the vertex positions defined by the red spheres is shown in gray.

5

1. Introduction• In previous work on 3D morphable models, the model is represented by

a mesh of control vertices, combined with an interpolation rule to create a continuous surface from the mesh.

• In contrast, Cashman assumes piecewise smooth objects, and use subdivision surfaces, which require many fewer parameters to describe.

• Based on the predefined topology, Cashman creates primary template shapes.

6

Images showing eight dolphins out of 32, from which we build an 8-parameter morphable model. Top: input images with silhouette annotations. Middle: The rigid dolphin prototype in initial position for our optimization, showing user-

provided point constraints (blue), each of which corresponds to a point in the image (shown in red). Bottom: final morphable model reconstruction overlaid on the input images.

1. Introduction

7

Cashman explicits surface representation parametrized by p control vertices represented by a vector and evaluated on a 2D domain .Ω

Let be the control vertices for the surface describing the object in image i.

2. System Input

� �∈ℝ3� 

 The surface is a mapping using where each point generates a 3D surface point denoted by ,which is linear in .

 

Xi

8

Image data: Silhouette (occluding contour)The silhouette is a 2D curve in the image plane, which is the projection of a possibly discontinuous 3D space curve.

2. System Input

Image data: Point constraints (user-specified)

10

Image data: Initial viewpointRepresent the camera by a similarity which transforms coordinates in model space so that each image projects to the xy-plane, with the viewing direction along the z-axis. The complete projection is:

Smoothness parametersTo complete this list of inputs, Cashman adds 2 scalar parameters and which control the smoothness of the model and of the deformation basis shapes.

2. System Input

0 def

11

The goal now is to define an objective function, the intuition of 3D model construction:

But is unknown. So, we need to use new parameter to make approximation.

We could build object function based on new parameters.

 

3. Energy Formulation

�̂→�  Min {} 

12

Overview of the parameters included in the formulation are show below:

We set the parameter to two group, P : pose parameter including U : preimage parameter U = ,

 

3. Energy Formulation

Reason shown in optimization part

12

13

Matching image silhouettes and constraints

To begin, assume we already have the contour preimage points, each of which corresponds to a silhouette sample. Therefore, the first energy term is the negative log probability of deviation from this equality under a Gaussian distribution:

Similar energy term for user-specified constraints:

For the surface to be consistent with the silhouette:

3. Energy Formulation

14

Only the fit of the points are not enough, we still need to make the surface consistent. To do that, the pare use the normal vector to represent the surface direction, which denote . After rotation with Matrix R, the normal to be exactly the. Thus, the paper presents the second energy function:

 

Z

� (�́��|� � )  ,0] 

3. Energy Formulation Normal Energy Term

14

15

Smoothness and regularization

3. Energy Formulation

we have enough information on the curve right now, however, we still need information for the surface. Then, the paper use linearized thin-plate energy to fit the assumption that the target function is smooth.

����= �́2

2 ∫Ω

❑

‖� �� (�́|�� )‖2+2‖��� (�́|�� )‖2

+‖� �� (�́|�� )‖2��́

 

The parameters should also be regularized 

�����=�∑

�=1

�

���2

 

16

Contour generatorshould be a continuous curve, and we want to prevent a result where the discrete points that we use to represent the contour preimage are arbitrarily scattered over.

 

�(�́�� ,�́� , �+1)� (¿)¿

����=�∑

�=1

��

¿

 

Contour generator continuity

3. Energy Formulation

¿�2 , h2

(¿)� (� )=min ¿

¿ 

16

17

3. Energy Formulation Complete energy function

+ 

This allows us to distinguish between the smoothness of the target object class andthe smoothness of the model

• Each term of the whole energy function can be written as a squared quantity !!

����= �́2

2 ∫Ω

❑

‖� �� (�́|�� )‖2+2‖��� (�́|�� )‖2

+‖� �� (�́|�� )‖2��́

 

17

18

4. Optimization Algorithm

Each term of the whole energy function can be written as a squared quantityParameters are continuous, we could use smooth-function optimizer to do the optimization

But, the results are not satisfied:

Initial estimate Only continuous local optimization

• the preimage is hard to do the initialization.

• Large volume of candidates and high dimension lead the problem easily to fall into a local minimum.

18

19

4. Optimization Algorithm

P : the pose parameter including

And U : the preimage parameter , while the U should be taken as “Hidden” parameter. AND, it is hard to estimate using local search method.

 

� ,�� (¿)

¿¿

min�

¿

�∗=���min�

¿

 

The paper search for the U with global discrete optimization only. Then, use the generated U to do the local classification to do the local optimization finding the P;

19

20

4. Optimization Algorithm

For the model (m = 0,1….D);

1: Set initialization of as well as other parameter;

2: Search the contour generator U with global optimization; is the candidate sets;

3. Do local continuous optimization starting from

Repeat the process D+1 times, the final result is

Use to construct model.

 

20

21

Global Search

4. Optimization Algorithm

Considering the , and are constant of The equation above could written as :

 

Thus, our target is to generate preimage points from candidates. 

The is the discrete candidate set including 10g+p candidate points.  

The target of search is to find the contour generator from the candidate set. 10 sampled uniformly

from inside each triangle

Each vertex

21

Ω Space

Candidate set:

22

Dynamic programing:

1:Starts with the shortest contour (including only one silhouette) to calculate the energy function for each candidate

2:Increase points one by one until achieve the required while still do the calculation for each candidate. Assume the bound is , the process would give a energy matrix.

3:Find the optimal preimage path by tracing back through the calculated cost matrix.

we have to make sure that the path forms a closed loop, while dynamic programming makes no such guarantee if used alone.

Using the constrains which are corresponded to the on the space we have.

Use branch-and-bound method to accelerate the process.

 

4. Optimization Algorithm

22

23

4. Optimization Algorithm Local Optimization

Each term of the whole energy function can be written as a squared quantity. So, all we need to do is to do is least squares minimization.The paper chooses to use the trust region-reflective algorithm provided by the MATLAB function lsqnonlin. The algorithm is implemented based on the following steps:

1:Formulate the two-dimensional trust-region subproblem:s1 is in the direction of the gradient g;s2 is either an approximate Newton direction or direction of negative

curvature 2:Solving s is linear combination of s1 and s2

3: If f(x + s)< f(x), then x = x + s.

4: Adjust Δ.

(��+1 ,��+1)=���min� ,�

�(� ,� ) 

23

24

5. Experiment Results

Silhouette Initial Model Resulting Model

Basic model

�� 

24

Thank you!

25