Globally Optimal Estimates for Geometric Reconstruction Problems

Globally Optimal Estimates for Geometric Reconstruction

ProblemsTom Gilat, Adi Lakritz

Advanced Topics in Computer Vision Seminar

Faculty of Mathematics and Computer Science

Weizmann Institute

3 June 2007

outline Motivation and Introduction

Background Positive SemiDefinite matrices (PSD) Linear Matrix Inequalities (LMI) SemiDefinite Programming (SDP)

Relaxations Sum Of Squares (SOS) relaxation Linear Matrix Inequalities (LMI) relaxation

Application in vision Finding optimal structure Partial relaxation and Schur’s complement

Motivation

Geometric Reconstruction Problems

Polynomial optimization problems (POPs)

Triangulation problem in L2 norm

points image theof worldin the source the,for estimation

pointes image ingcorrespond...

matrices camera 43 - ,...,

:Given

Multi view - optimization

2-views – exact solution

perspective camera i

iii XPx

ix' err

points image theof worldin the source the,for estimation

pointes image ingcorrespond...

matrices camera 43 - ,...,

:Given

minimize reprojection error in all cameras

convexnon - ))(,()err( :functionerror 1

iii XPxdXL

convex - 0

cameras all offront in points all isdomain

0 subject to )err(min i X Polynomial

minimization problem

Non convex

More computer vision problems

• Reconstruction problem:known cameras, known corresponding pointsfind 3D points that minimize the projection error of given image points– Similar to triangulation for many points and cameras

• Calculating homographygiven 3D points on a plane and corresponding image points, calculate homography

• Many more problems

Optimization problems

Introduction to optimization problems

m 1 :f ,x

sconstraint– m 1 ,b (x) subject to

function objective– (x) minimize

set feasiblein vectorsall amongsmallest is )~(

set feasible~ if optimal is ~

sconstraint esatisfy th that vectorsall set feasible

optimization problems

0 s.t. }1{),...,( thereareor

d?partitione be ,..., sequence can the

?0 }))1(()()({ min 222 i

i xxaxf

NP - complete

optimization

convex non convex

Linear

Programming

SemiDefinite

Programming

solutions exist:

interior point methods

problems: local optimum

or high computational cost

non convex optimization

level curves of f

Many algorithms

Get stuck in

local minima

MinMax

Non convex feasible set

optimization

convex non convex

LP SDP

solutions exist:

interior point methods

problems: local optimum

or high computational cost

global optimization – algorithms that converge to optimal solution

relaxation of problem

positive semidefinite (PSD) matrices

0)()(: 2 xAxAxAxAAxMxxAAM TTTTTTTT

0 MxxRx Tn

0by denoted M

Definition: a matrix M in Rn×n is PSD if

1. M is symmetric: M=MT

2. for all

M can be decomposed as AAT (Cholesky)

Proof:

s)eigenvalue enonnegativ and theorm(spectral :

positive semidefinite (PSD) matrices

0 MxxRx Tn

0by denoted M

Definition: a matrix M in Rn×n is PSD if

1. M is symmetric: M=MT

2. for all

M can be decomposed as AAT (Cholesky)

nT RvvvM , then 1rank is M if

principal minors

The kth order principal minors of an n×n symmetric matrix M are the

determinants of the k×k matrices obtained by deleting n - k rows and

the corresponding n - k columns of M

first order: elements on diagonal

second order:

333231

232221

131211

333231

232221

131211

diagonal minors

second order:

333231

232221

131211

333231

232221

131211

diagonal minors

second order:

333231

232221

131211

333231

232221

131211

diagonal minors

enonnegativ are M of minors principal theall iff 0 Mmatrix A

third order: det(M)

second order:

333231

232221

131211

333231

232221

131211

Set of PSD matrices in 2D

yx2 ,0, yxzzx

Set of PSD matrices

This set is convex

00)(00,0 tMxMxtMxxtM TT

LMxLMxx

LxxMxxxLMT

0)1( so LttM

Proof:

LMI – linear matrix inequality

convex is }0|{Kset feasible A(x)Rx n

)(xAnR

matrices symmetrick k are A ,R)x,...,(xx

iii AxAxA

matricesk k

KyttxyttxA

yAtxtA

A(x),A(y)tKx,y

)1(0))1((

0)()1()(

:Proof

example: find the feasible set of the 2D LMI

)(xA2A1A0A

reminder

enonnegativ are M of minors principal theall iff 0 Mmatrix A

01 ,02 ,01 221 xxx

1st order principal minors

)(xA2A1A0A

2A1A0A

0)1 )(1(

,0)1 )(2(

,0)()2 )(1(

2nd order principal minors

2A1A0A

))()2 )(1)((1(

221211

3rd order principal minors

Intersection of all inequalities

Semidefinite Programming (SDP) = LMI

an extension of LP

0)( tosub.x minimize :SDP T xb, BAxc

bAxxcT tosub. minimize :LP

Sum Of Squares relaxation (SOS)

Unconstrained polynomial optimization problem (POP)

means the feasible set is Rn

H. Waki, S. Kim, M. Kojima, and M. Muramatsu. SOS and SDP relaxations for POPs with structured sparsity. SIAM J. Optimization, 2006.

Sum Of Squares relaxation (SOS)

}polynomial a is )( and 0)(|{

:Define

xfRxxffN n

})()( s.t. )(),...,( spolynomial|{1

iin xgxfxgxgfSOS

NSOSNSOS ,

rare is \)( SOSNxf

SOS relaxation for unconstrained polynomials

Npxfp )( s.t. max find :P')(min find :P xf

)(min is 0)(s.t max so

holdnot will thisn larger thay any for and

then)(min if

:Proof

x fpx fp

SOS relaxation for unconstrained polynomials

Npxfp )( s.t. max find :P')(min find :P xf

relaxation - )( s.t. max find :'P' SOSqxfq

SDPby solves becan 'P'

: on bound guarantees qpp

monomial basis

rnd(r)xvr ))(dim(

T,x,xx,x,x,x

xxx-xx-xf(x)

)1)(6,4,5,3,2,1(

65432122

212121

then degree of is if )(rdr

T Ra(x) where vaf(x)rf(x) example

SOS relaxation to SDP

}0)()({

})()({

}|))(({

}))(deg( )({

2r degree of spolynomial SOS ofset

| VxVvxv

R | axvaaxv

rxg|xg

0V s.t. )()()( tosub. max xVvxvqxfq rT

SOS relaxation to SDP

666564636261

565554535251

464544434241

363534333231

262524232221

161514131211

VVVVVV

0V 0,V else ,VV24 ,2V3- ,2V2 ,V-

tosub. max

ii5546131211 q

2121 432 xxx-xf(x) example:

SOS for constrained POPs

possible to extend this method for constrained POPs

by use of generalized Lagrange dual

SOS relaxation summary

So we know how to solve a POP that is a SOS

And we have a bound on a POP that is not an SOS

H. Waki, S. Kim, M. Kojima, and M. Muramatsu. SOS and SDP relaxations for POPs with structured sparsity. SIAM J. Optimization, 2006.

POP SOS

problem

relaxationGlobal

estimate

relaxations

POP SOS

problem

relaxationGlobal

estimate

POP linear & LMI

problem

relaxationGlobal

estimate

converge

LMI relaxations

Constraints are handled

Convergence to optimum is guaranteed

Applies to all polynomials, not SOS as well

A maximization problem

Note that:a.Feasible set is non-convex. b.Constraints are quadratic

Feasible set

023)(s.t.

xxxxxg

LMI – linear matrix inequality, a reminder

convex is }0|{Kset feasible A(x)Rx n

)(xAnR

matrices symmetrick k are A ,R)x,...,(xx

iii AxAxA

matricesk k

An SDP:

Motivation

0...s.t.

Polynomial Optimization

Problem

SDP with solution close

to global optimum of the

original problem

What is it good for?

SDP problems can be solved much more efficiently then general

optimization problems.

Linear + LMI + rank constraints

LMI Relaxations is iterative process

Step 1: introduce new variables

Step 2: relax constraints

Apply higher

relaxations

LMI relaxation – step 1

Replace monomials by “lifting variables”ji xx 21

ijji yxx 21

Example:

0)(0)( 110110221212 yyyxgxxxxxg

(the R2 case)

023)(s.t.

xxxxxg

023)(s.t.

1101102

0220011

Introducing lifting variables

Lifting

023)(s.t.

1101102

0220011

Not equivalent to the original problem.

Lifting variables are not independent in the original problem:

New problem is linear, in particular convex

023)(s.t.

1101102

0220011

2111201110 xxyxyxy

100111 yyy

Goal, more specifically

Linear problem

(obtained by lifing)

“relations constraints” on lifting variables

023)(s.t.

1101102

0220011

) :demand weexample,(For 100111 yyy

Relaxation

Question: how do we guarantee that the relations between lifting variables hold?

on..... so and101020

100111

LMI relaxation – step 2

021101

112010

MApply lifting and get:

Take the basis of the degree 1 polynomials.

Note that:

Txxxv ],,1[)( 211

)()(22212

xvxv T

Because: te)semidefini positive is(0 AAvvA T

If the relations constraints hold then

021101

112010

This is because we can decompose M as follows:

],,1[],,1[

01100110

021101

112010

on...) so and ( 110110 yyy Assuming relations hold

Rank M = 1

We’ve seen:

Relations constraints

021101

112010

What about the opposite:

021101

112010

MRelations constraints

1Mrank

This is true as well

By the following:

LMI relaxation – step 2, continued

021101

112010

MRelations constraints

1Mrank

1 and0 MRankMvvM T

All relations equalities are in the set

of equalities ijT

ij vvM ][][

Conclusion of the analysis

023)(s.t.

1101102

0220011

021101

112010

1,0 MrankM The “y feasible set”Subset of feasible set with

Relations constraints hold

Original problem is equivalent to the following:

together with the additional constraint

023)(s.t.

1101102

0220011

021101

112010

Relaxation, at last

We denote moment matrix

of order 1

1)(1 yMrank

Relax by dropping the non-convex constraint 1)(1 yMrank

LMI relaxation of order 1

Feasible set

023)(s.t.

1101102

0220011

Rank constrained LMI vs. unconstrained

LMI relaxations of higher order

It turns out that we can do better:

Apply LMI relaxations of higher order

A tighter SDP

Relaxations of higher order incorporate the inequality constraints in LMI

• We show relaxation of order 2

• It is possible to continue and apply relaxations

• Theory guarantees convergence to global optimum

Let be a basis of polynomials

of degree 2.

Again,

Lifting gives:

Again, we will relax by dropping the rank constraint.

LMI relaxations of second order

Txxxxxxxv ],,,,,1[)( 2221

0)()( 22 Txvxv

Inequality constraints to LMIReplace our constraints by LMIs and have a tighter relaxation.

Linear Constraint : 01)( 213 xxxg 01)( 113 yygLifting

LiftingLMI Constraint : 0)()()( 113 Txvxvxg 0)( 31 ygM

For example, 211010111031231 )1()()( yyyyyygygM

This procedure brings a new SDP

LMI relaxations of order 2

Second SDP feasible set is included in the first SDP feasible set

Silimarly, we can continue and apply higher order relaxations.

023)(s.t.

1101102

0220011

,0)(,0)(,0)(s.t.

312111

ygMygMygM

If the feasible set defined by constraints is compact, then under mild additional assumptions, Lassere proved in 2001 that there is an asymptotic convergence guarantee:

0)( xgi

gpkklim

kp is the solution to k’th relaxation

g is the solution for the original problem (finding a maximum)

Theoretical basis for the LMI relaxations

Moreover, convergence is fast: kp is very close to g for small k

Lasserre J.B. (2001) "Global optimization with polynomials and the problem of moments" SIAM J. Optimization 11, pp 796--817.

The method provides a

certificate of global optimality:

An important experimental observation:

SDP solution is global

optimum

Minimizing

Low rank moment matrix

Checking global optimality

1)( yMrank n

))(( yMtrace n

We add to the objective function the trace of the moment matrix weighted by a sufficiently small positive scalar

,0)(,0)(,0)(s.t.

))((max

312111

ygMygMygM

yMtracey

LMI relaxations in vision

Application

point measured theis ~upoint dreprojecte theisu

Camera center

Image plane

Finding optimal structure A perspective camera

P is the camera matrix and is the depth.

Measured image points are corrupted by independent Gaussian noise.

We want to minimize the least squares errors between measured and projected points.

The relation between a U in the 3D space and u in the image plane is given by:

Niui ..1,~

0)(..))(,~(min1

xtsxuud i

We therefore have the following optimization problem:

is the Euclidean distance.

x is the set of unknowns

)()())(,~(

xfxfxuud

)(),(),( 21 xxfxf iii Where are polynomials.

Each term in the cost function can be written as:

Our objective is therefore to minimize a sum of rational functions.

How can we turn the optimization problem into a polynomial optimization problem?

Suppose that each term in

iii xuud

2))(,~( has an upper bound , theni

Then our optimization problem is equivalent to the following:

xxfxfts

..10)(

)()()(..

...min22

This is a polynomial optimization problem, for which we apply LMI relaxations.

Note that we introduced many new variables – one for each term.

Problem: an SDP with a large number of variables can be computationally demanding.

A large number of variables can arise from:

LMI relaxations of high order

Introduction of new variables as we’ve seen

This is where partial relaxations come in.

For that we introduce Schur’s complement.

Partial Relaxations

Schur’s comlement

BABC T

Schur’s comlement - applying

)()()( 222

0)()()(

Derivation of right side:C - BT * A-1 * B > 0

Schur’s complement allows us to state our optimization problem as follows:

..10)(

...min

Partial relaxations

The only non-linearity is due to 2)(xi

We can apply LMI relaxations only on Txxx ,...],[ 21 and leave Nii ..1,

If we were to apply full relaxations for all variables, the problem would become

Intractable for small N.

Partial relaxations

Disadvantage of partial relaxations: we are not able to ensure asymptotic convergence to the global optimum.

However, we have a numerical certificate of global optimality just as in the case of full relaxations:

The moment matrix of the relaxed variables is of rank one

Solution of partially relaxed problem is the global optimum

Full relaxation vs. partial relaxtion

Application: Triangulation, 3 cameras

Goal: find the optimal 3D point. Camera matrices are known, measured point is assumed to be in the origin of each view.

Camera matrices:

Summary• Geometric Vision Problems to POPs

– Triangulation and reconstruction problem

• Relaxations of POPs– Sum Of Squares (SOS) relaxation

• Guarantees bound on optimal solution

• Usually solution is optimal

– Linear Matrix Inequalities (LMI) relaxation• First order LMI relaxation: lifting, dropping rank constraint

• Higher order LMI relaxation: linear constraints to LMIs

• Guarantee of convergence, reference to Lassere

• Certificate of global optimality

• Application in vision• Finding optimal structure • Partial relaxation and Schur’s complement• Triangulation problem, benefit of partial relaxations

References F. Kahl and D. Henrion. Globally Optimal Estimates for

Geometric Reconstruction Problems. Accepted IJCV H. Waki, S. Kim, M. Kojima, and M. Muramatsu. Sums of

squares and semidefinite programming relaxations for polynomial optimization problems with structured sparsity. SIAM J. Optimization, 17(1):218–242, 2006.

J. B. Lasserre. Global optimization with polynomials and the problem of moments. SIAM J. Optimization, 11:796–817, 2001.

S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.

R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, 2004. Second Edition.

Globally Optimal Estimates for Geometric Reconstruction Problems

Documents

Transcript of Globally Optimal Estimates for Geometric Reconstruction Problems

Geometric Design Tables (New Construction/ Reconstruction) 3...INDIANA DEPARTMENT OF TRANSPORTATION—2013 DESIGN MANUAL CHAPTER 53 Geometric Design Tables (New Construction/ Reconstruction)

3D Object Reconstruction From Hand-Object Interactions€¦ · captured hand motion together with texture and geometric features for object reconstruction as in Figure 3. Since the

Geometric Reconstruction in Bioluminescence Tomography (BLT)mtm.ufsc.br/~aleitao/public/slides-cbm29/slides-rieder.pdf · 2013-07-17 · Geometric Reconstruction in Bioluminescence

A bio-inspired geometric model for sound reconstruction › ~lus219 › publis › Sound_reconstruction.pdfA bio-inspired geometric model for sound reconstruction Ugo Boscain1, Dario

On Differential Photometric Reconstruction for Unknown ...vision.ucsd.edu/sites/default/files/secondordertheory.pdf · exact priors for a full geometric reconstruction. These results

3D Scene Reconstruction by Integration of Photometric and … · Well known geometric approaches to 3D reconstruction include passive methods such as Stereo, Structure from Motion

4.3 Geometric Algorithms for 3D Interface Reconstructionshashkov/papers/ahn_geometry.pdf · 4.3 Geometric Algorithms for 3D Interface Reconstruction 407 Fig. 1. Generalized polyhedral

Graphics - parasol.tamu.eduwelch/teaching/181.s08/schaefer.pdf · Geometric Modeling Deformation and Animation Surface reconstruction Scientific visualization John Keyser Robust Geometric

LIDAR-based geometric reconstruction of boreal type forest ... · LIDAR-based geometric reconstruction of boreal type forest stands at single tree level for forest and wildland fire

New geometric interpretation and analytic solution for quadrilateral reconstruction (ICPR-2014)

Reconstruction and Geometric Algorithmsmaverick.inria.fr/~Romain.Vergne/reconstruction/Lecture5.pdf · Curve Reconstruction Algorithm (B) Traversal of triangles and construction of

geometric Design Tables (new Construction/ Reconstruction) · Geometric Design Tables (New Construction/ Reconstruction) Design ... 53-1 Geometric Design Criteria for Freeway ...

A geometric convection approach of 3-D reconstruction · 2004-05-18 · A geometric convection approach of 3-D reconstruction Raphaºlle Chaine Prisme Project, INRIA Sophia-Antipolis,

Geometric Design of Existing Freeway (3R) or (4R) … DEPARTMENT OF TRANSPORTATION—2013 DESIGN MANUAL CHAPTER 54 Geometric Design of Existing Freeway (3R) or (4R) Partial Reconstruction

A geometric convection approach of 3-D reconstruction · A geometric convection approach of 3-D reconstruction Raphaºlle Chaine Prisme Project, INRIA Sophia-Antipolis, France Abstract

Globally Optimal Estimates for Geometric Reconstruction Problems Tom Gilat, Adi Lakritz Advanced Topics in Computer Vision Seminar Faculty of Mathematics.

RECONSTRUCTION OF ERODED AND VISUALLY COMPLICATED … · 2020. 9. 18. · RECONSTRUCTION OF ERODED AND VISUALLY COMPLICATED ARCHAEOLOGICAL GEOMETRIC PATTERNS: MINARET CHOLI, IRAQ

Geometric reconstruction methods for electron tomography · Geometric tomography [13], for instance, is concerned in part with the tomographic reconstruction of homogeneous (i.e.,

Image metrics for geometric calibration of a cone-beam ... · The precise knowledge of the geometric acquisition parameters directly inﬂuences the quality of a tomographic reconstruction.

On Differential Photometric Reconstruction for Unknown, Isotropic …vision.ucsd.edu/~manu/pdf/pami12_secondordertheory.pdf · exact priors for a full geometric reconstruction. These