Computer Visioncmput 498/613

3D Modeling from images3D Modeling from images

Martin JagersandMartin Jagersand

Pinhole camera• Central projectionCentral projection

• Principal point & aspectPrincipal point & aspect

10100

000

000

~

1

)/,/(),,(

Z

Y

X

f

f

Z

fY

fX

y

x

ZfYZfXZYX TT

1100

10

01

1

1

1

~

1

y

x

cp

cp

cyp

cxp

v

u

yy

xx

yy

xx c

px

py

The projection matrix:

cam

camyy

xx

IK

cp

f

cp

f

X0x

Xx

]|[

0100

00

00

Projective camera

• Camera rotation and translationCamera rotation and translation

• The projection matrixThe projection matrix XtXXtX TT

camcam RRR

Xtx IKRT

PIn general:

•P is a 3x4 matrix with 11 DOF

•Finite: left 3x3 matrix non-singular

•Infinite: left 3x3 matrix singular

Properties: P=[M p4]

•Center:

•Principal ray (projection direction)

0,01

0

41

dd

Cp

C

C

MM

P

3)det( mv M

• Infinite cameras where the last row of Infinite cameras where the last row of PP is is (0,0,0,1)(0,0,0,1)• Points at infinity are mapped to points at infinityPoints at infinity are mapped to points at infinity

Affine cameras

z

z

y

x

Ty

x

td

t

t

t

R

d

t

t

KP

0

0

t

k

j

i

0

j

i

td0

j

k

i{camera}

Z

X

Y{world}

x0

• ErrorError)( 0

0

xxxx

projperspaff d

Good approximation:

• small compared to d0

• point close to principal ray

X

X’

xaff

xpersp

Camera calibration

• 11 DOF => at least 6 points11 DOF => at least 6 points• Linear solutionLinear solution

– Normalization requiredNormalization required– Minimizes algebraic errorMinimizes algebraic error

• Nonlinear solutionNonlinear solution– Minimize geometric error (pixel re-projection)Minimize geometric error (pixel re-projection)

• Radial distortionRadial distortion– Small near the center, increase towards peripherySmall near the center, increase towards periphery

1

0min

p

pA

ii P Xx known?known

...1 221 rKrKr

Application: raysets

Gortler and al.; Microsoft Lumigraph

t

v

s

u (s,t)

(u,v)

H-Y Shum, L-W He; MicrosoftConcentric mosaics

Ck

Cl C0

Li

Lj

vi

vj i j

• Projection equationProjection equation

xxii=P=PiiXX

• Resection:Resection:– xxii,X P,X Pii

Multi-view geometry - resection

Given image points and 3D points calculate camera projection matrix.

• Projection equationProjection equation

xxii=P=PiiXX

• Intersection:Intersection:– xxii,P,Pi i XX

Multi-view geometry - intersection

Given image points and camera projections in at least 2 views calculate the 3D points (structure)

• Projection Projection equationequation

xxii=P=PiiXX

• Structure from Structure from motion (SFM)motion (SFM)– xxii P Pii,, XX

Multi-view geometry - SFM

Given image points in at least 2 views calculate the 3D points (structure) and camera projection matrices (motion)

•Estimate projective structure

•Rectify the reconstruction to metric (autocalibration)

Depth from stereo

•Calibrated aligned camerasCalibrated aligned cameras

Z

X(0,0) (d,0)

Z=f

xl xrrl

rl

xx

dfZ

dXx

fX

x

fZ

)(

Disparity d

Trinocular Vision System

(Point Grey Research)

Application: depth based reprojection

3D warping, 3D warping, McMillanMcMillan

Plenoptic modeling, Plenoptic modeling, McMillan & BishopMcMillan & Bishop

Application: depth based reprojection

Layer depth images,Layer depth images, Shade et al. Shade et al.

Image based objects,Image based objects, Oliveira & Bishop Oliveira & Bishop

Affine camera factorization3D structure from many images

The affine projection equations are The affine projection equations are

1

j

j

j

yi

xi

ij

ij

Z

Y

X

P

Py

x

10001

1

j

j

j

yi

xi

ij

ij

Z

Y

X

P

P

y

x

~

~

4

4

j

j

j

yi

xi

ij

ij

yiij

xiij

Z

Y

X

P

Py

x

Py

Px

Orthographic factorization

The ortographic projection equations are The ortographic projection equations are

where where njmijiij ,...,1,,...,1,Mm P

All equations can be collected for all All equations can be collected for all ii and and jj

wherewhere

n

mmnmm

n

n

M,...,M,M,,

mmm

mmm

mmm

212

1

21

22221

11211

M

P

P

P

Pm

MPm

M ~

~m

j

j

j

jyi

xi

iij

ijij

Z

Y

X

,P

P,

y

xP

Note that P and M are resp. 2mx3 and 3xn matrices and

therefore the rank of m is at most 3

(Tomasi Kanade’92)

Orthographic factorization

Factorize Factorize mm through singular value decomposition through singular value decomposition

An affine reconstruction is obtained as followsAn affine reconstruction is obtained as follows

TVUm

TVMUP ~,

~

(Tomasi Kanade’92)

n

mmnmm

n

n

M,...,M,M

mmm

mmmmmm

min 212

1

21

22221

11211

P

P

P

Closest rank-3 approximation yields MLE!

0~~

1~~

1~~

TT1

TT1

TT1

yi

xi

yi

yi

xi

xi

PP

PP

PP

QQ

QQ

QQ

0~~

1~~

1~~

T

T

T

yi

xi

yi

yi

xi

xi

PP

PP

PP

C

C

C

A metric reconstruction is obtained as followsA metric reconstruction is obtained as follows

Where A is computed from Where A is computed from

Orthographic factorization

Factorize Factorize mm through singular value decomposition through singular value decomposition

An affine reconstruction is obtained as followsAn affine reconstruction is obtained as follows

TVUm

TVMUP ~,

~

MQMQPP~

,~ 1

0

1

1

T

T

T

yi

xi

yi

yi

xi

xi

PP

PP

PP 3 linear equations per view on

symmetric matrix C (6DOF)

Q can be obtained from C through Cholesky factorisation

and inversion

(Tomasi Kanade’92)

N-view geometryAffine factorization

[Tomasi &Kanade ’92][Tomasi &Kanade ’92]

•Affine camera Affine camera

•ProjectionProjection

•nn points, points, mm views: measurement matrix views: measurement matrix

]|[ tMP M 2x3 matrix; t 2D vector

t

Z

Y

X

My

x

nmm

nm

n

M

M

W XX

xx

xx

1

1

1

111

~~

~~

W: Rank 3XMVDUW

UDVWT

nm

T

ˆˆˆ33332

Assuming isotropic zero-mean Gaussian noise, factorization achieves ML affine reconstruction.

txx ~

Weak perspective factorization

[D. Weinshall][D. Weinshall]

•Weak perspective cameraWeak perspective camera

•Affine ambiguityAffine ambiguity

•Metric constraintsMetric constraints

j

i

s

sM

)ˆ)(ˆ(ˆˆˆ 11 XQQMXQQMW

0ˆˆ

ˆˆˆˆ 2

ji

jjii

sQQs

ssQQssQQsTT

TTTT

Extract motion parameters Eliminate scale Compute direction of camera axis k = i x j parameterize rotation with Euler angles

Full perspective factorization

The camera equations The camera equations

for a fixed image for a fixed image ii can be written in matrix form can be written in matrix form asas

where where

njmijiijij ,...,1,,...,1,Mmλ P

MPm iii

imiii

mimiii mmm

λ,...,λ,λdiag

M,...,M,M,,...,,

21

2121

Mm

Perspective factorization

All equations can be collected for all All equations can be collected for all ii as as

wherewherePMm

mnn P

P

P

P

m

m

m

m...

,...

2

1

22

11

In these formulas In these formulas mm are known, but are known, but ii,,PP and and MM are unknown are unknown

Observe that Observe that PMPM is a product of a 3 is a product of a 3mmx4 matrix and a 4xx4 matrix and a 4xnn matrix, i.e. it is a rank 4 matrixmatrix, i.e. it is a rank 4 matrix

Perspective factorization algorithm

Assume that i are known, then PM is known.

Use the singular value decomposition PM=U VT

In the noise-free case

S=diag(1,2,3,4,0, … ,0)and a reconstruction can be obtained by setting:

P=the first four columns of U.M=the first four rows of V.

Iterative perspective factorization

When i are unknown the following algorithm can be used:

1. Set ij=1 (affine approximation).

2. Factorize PM and obtain an estimate of P and M. If 5 is sufficiently small then STOP.

3. Use m, P and M to estimate i from the camera equations (linearly) mi i=PiM

4. Goto 2.

In general the algorithm minimizes the proximity measure P(,P,M)=5Note that structure and motion recovered

up to an arbitrary projective transformation

Projective factorizationsummary

[Sturm & Triggs’96][ Heyden ‘97 ][Sturm & Triggs’96][ Heyden ‘97 ]

•Measurement matrixMeasurement matrix

•Known projective depthKnown projective depth

– Projective ambiguity Projective ambiguity

• Iterative algorithmIterative algorithm– Reconstruct withReconstruct with– Reestimate depth and iterate Reestimate depth and iterate

nmm

nmn

mm

nn

P

P

W XX

xx

xx

1

1

11

1111

11

3mxn matrix

Rank 4

ij

XPVDUW

UDVWT

nm

T

ˆˆˆ44442

1ij

ij

Further Factorization work

Factorization with uncertaintyFactorization with uncertainty

Factorization for dynamic scenesFactorization for dynamic scenes

(Irani & Anandan, IJCV’02)

(Costeira and Kanade ‘94)

(Bregler et al. 2000, Brand 2001)

Sequential 3D structure from motionusing 2 and 3 view geom

• Initialize structure and motion from two viewsInitialize structure and motion from two views• For each additional viewFor each additional view

– Determine poseDetermine pose– Refine and extend structureRefine and extend structure

• Determine correspondences robustly by jointly Determine correspondences robustly by jointly estimating matches and epipolar geometry estimating matches and epipolar geometry

Images

2 view geometry (Epipolar geometry)

Fundamental matrix

• Algebraic representation of epipolar geometryAlgebraic representation of epipolar geometry

Step 1: X on a plane

Step 2: epipolar line l’xxe

xexel

xx

FH

H

]'[

']'['''

'

0' xx FT

F

•3x3, Rank 2, det(F)=0

•Linear sol. – 8 corr. Points (unique)

•Nonlinear sol. – 7 corr. points (3sol.)

•Very sensitive to noise & outliers

[Faugeras ’92, Hartley ’92 ]

Epipolar lines:

Epipoles:

Projection matrices:

'|']'['

]|[

0'0

''

evee

0

ee

xlxl

T

T

T

FP

IP

FF

FF

3,4,N view geometry

• Trifocal tensor (3 view geometry)Trifocal tensor (3 view geometry)

],,[: 321 TTTT 3x3x3 tensor;

27 params. (18 indep.)

0]"[)(]'[

"]3,2,1['

xxx

lll

ii

i

T

T

TTT lines

points

• Quadrifocal tensor (4 view geometry) Quadrifocal tensor (4 view geometry) [Triggs ’95]

•Multiview tensors Multiview tensors [Hartley’95][ Hayden ‘98]

There is no additional constraint between more than 4 images. All the constraints There is no additional constraint between more than 4 images. All the constraints can be expressed using F,triliear tensor or quadrifocal tensor. can be expressed using F,triliear tensor or quadrifocal tensor.

[Hartley ’97][Torr & Zisserman ’97][ Faugeras ’97]

Initial structure and motion

eeaFeP

0IPT

x2

1

Epipolar geometry Projective calibration

012 FmmT

compatible with F

Yields correct projective camera setup(Faugeras´92,Hartley´92)

Obtain structure through triangulation

Use reprojection error for minimizationAvoid measurements in projective space

Compute Pi+1 using robust approachFind additional matches using predicted projectionExtend, correct and refine reconstruction

2D-2D

2D-3D 2D-3D

mimi+1

M

new view

Determine pose towards existing structure

Non-sequential image collections

4.8im/pt64 images

3792

po

ints

Problem:Features are lost and reinitialized as new features

Solution:Match with other close views

For every view iExtract featuresCompute two view geometry i-1/i and matches Compute pose using robust algorithmRefine existing structureInitialize new structure

Relating to more views

Problem: find close views in projective frame

For every view For every view iiExtract featuresExtract featuresCompute two view geometry Compute two view geometry ii-1/-1/ii and matches and matches Compute pose using robust algorithmCompute pose using robust algorithmFor all For all closeclose views views kk

Compute two view geometry Compute two view geometry kk//ii and matches and matchesInfer new 2D-3D matches and add to listInfer new 2D-3D matches and add to list

Refine pose using all 2D-3D matchesRefine pose using all 2D-3D matchesRefine existing structureRefine existing structureInitialize new structureInitialize new structure

Determining close views

• If viewpoints are If viewpoints are closeclose then most image changes then most image changes can be modelled through a can be modelled through a planar homographyplanar homography

• Qualitative distance measureQualitative distance measure is obtained by is obtained by looking at the looking at the residual errorresidual error on the on the best possible best possible planar homographyplanar homography

Distance = m´,mmedian min HD

9.8im/pt

4.8im/pt

64 images

64 images

3792

po

ints

2170

po

ints

Non-sequential image collections (2)

Refining structure and motion

• Minimize reprojection errorMinimize reprojection error

– Maximum Likelyhood Estimation Maximum Likelyhood Estimation (if (if error zero-mean Gaussian noise)error zero-mean Gaussian noise)

– Huge problem but can be solved efficientlyHuge problem but can be solved efficiently(Bundle adjustment)(Bundle adjustment)

m

k

n

iikD

ik 1 1

2

kiM̂,P̂

M̂P̂,mmin

Refining a captured model:Bundle adjustment

• Refine structure XRefine structure Xjj and motion P and motion Pii

• Minimize geometric errorMinimize geometric error• ML solution, assuming noise is GaussianML solution, assuming noise is Gaussian• Tolerant to missing dataTolerant to missing data

ji

ijj

iPd,

2),ˆˆ(min xX

Projective ambiguity andself-calibration

Given an uncalibrated image sequence with corresponding point it is possible to reconstruct the object up to an unknown projective transformation

• Autocalibration (self-calibration): Determine a projective transformation T that upgrades the projective reconstruction to a metric one.

T

´M´M))((Mm 1 PTPTP

Remember:Stratification of geometry

15 DOF 12 DOFplane at infinity

parallelism

More general

More structure

Projective Affine Metric

7 DOFabsolute conicangles, rel.dist.

Constraints ?

• Scene constraintsScene constraints– Parallellism, vanishing points, horizon, ...Parallellism, vanishing points, horizon, ...– Distances, positions, angles, ...Distances, positions, angles, ...

Unknown scene no constraints

• Camera extrinsics constraints–Pose, orientation, ...

Unknown camera motion no constraints • Camera intrinsics constraints

–Focal length, principal point, aspect ratio & skew

Perspective camera model too general some constraints

Euclidean projection matrix

tRRKP TT

1yy

xx

uf

usf

K

Factorization of Euclidean projection matrix

Intrinsics:

Extrinsics: t,R

Note: every projection matrix can be factorized,

but only meaningful for euclidean projection matrices

(camera geometry)

(camera motion)

Constraints on intrinsic parameters

Constant Constant e.g. fixed camera:e.g. fixed camera:

KnownKnowne.g. rectangular pixels:e.g. rectangular pixels:

square pixels:square pixels: principal point known:principal point known:

21 KK

0s

1yy

xx

uf

usf

K

0, sff yx

2,

2,

hwuu yx

Self-calibration

Upgrade from Upgrade from projectiveprojective structure to structure to metricmetric structure using structure using constraintsconstraints onon intrinsicintrinsic camera parameters camera parameters– Constant intrinsicsConstant intrinsics

– Some known intrinsics, others varyingSome known intrinsics, others varying

– Constraints on intrincs and restricted motionConstraints on intrincs and restricted motion(e.g. pure translation, pure rotation, planar motion)(e.g. pure translation, pure rotation, planar motion)

(Faugeras et al. ECCV´92, Hartley´93,

Triggs´97, Pollefeys et al. PAMI´98, ...)

(Heyden&Astrom CVPR´97, Pollefeys et al. ICCV´98,...)

(Moons et al.´94, Hartley ´94, Armstrong ECCV´96, ...)

A counting argument

• To go from projective (15DOF) to metric (7DOF) at least 8 To go from projective (15DOF) to metric (7DOF) at least 8 constraints are neededconstraints are needed

• Minimal sequence length should satisfyMinimal sequence length should satisfy

• Independent of algorithmIndependent of algorithm• Assumes general motion (i.e. not critical)Assumes general motion (i.e. not critical)

8#1# fixednknownn

Conics•Conic: Conic:

– Euclidean geometry: hyperbola, ellipse, parabola & degenerateEuclidean geometry: hyperbola, ellipse, parabola & degenerate– Projective geometry: equivalent under projective transformProjective geometry: equivalent under projective transform– Defined by 5 pointsDefined by 5 points

•TangentTangent

•Dual conic C*Dual conic C*

0

022

xx C

feydxcybxyaxT

fed

ecb

dba

C

2/2/

2/2/

2/2/

xl C

0* ll CT

Quadrics

Quadrics: Quadrics: QQ 4x4 symmetric matrix 4x4 symmetric matrix

9 DOF (defined by 9 points in general pose)9 DOF (defined by 9 points in general pose)

•Dual: Dual: Q*Q*Planes tangent to the quadricPlanes tangent to the quadric

0XX QT

0* ππ QT

Summary:Conics & Quadrics

0mmT C 0ll *T C

1* CC

conics

0MMT Q 0*T Q

1* QQ

quadrics

1T´~ CHHCC

T*** ´~ HHCCC T´ TTQ~QQ ***

1T´~ QTTQQ

transformations

T** ~ PPQC

projection

The absolute conic

• Absolute conic is a imaginary circle onAbsolute conic is a imaginary circle on• The absolute dual quadric (rim quadric)The absolute dual quadric (rim quadric)

• In a metric frameIn a metric frame

π

0),,(),,(

0

)1,0,0,0(

321321

4

23

22

21

TxxxIxxx

x

xxx

π

*

0

0*

*

T

T

I

0

0

Note: is the nullspace of

*

On : π

*

Fixed under similarity transf.

Self-calibration

•Theoretically formulated by Theoretically formulated by [Faugeras ’92][Faugeras ’92]

•2 basic approaches2 basic approaches– Stratified: recover Stratified: recover – Direct: recover Direct: recover [Triggs’97][Triggs’97]

•Constraints:Constraints:– Camera internal constraintsCamera internal constraints

– Constant parameters Constant parameters [Hartley’94][ Mohr’93][Hartley’94][ Mohr’93]

– Known skew and aspect ratioKnown skew and aspect ratio [Hayden&[Hayden&ÅÅstrströöm’98][Pollefeys’98]m’98][Pollefeys’98]

– Scene constraints (angles, ratios of length)Scene constraints (angles, ratios of length)

•Choice of H:Choice of H: Knowing camera Knowing camera KK and and

π

*

TTT K

KH )1,(,

1p

p

0

π

Absolute Dual Quadric and Self-calibration

Eliminate extrinsics from equationEliminate extrinsics from equation

Equivalent to projection of dual quadric

))(Ω)((Ω *1* TTTTT PTTTPTPPKK

Abs.Dual Quadric also exists in projective world

T´Ω´´ * PP Transforming world so thatreduces ambiguity to metric

** ΩΩ´

*

*

projection

constraints

Absolute conic = calibration object which is always present but can only be observed through constraints on the intrinsics

Tii

Tiii Ωω KKPP

Absolute Dual Quadric and Self-calibration

Projection equation:Projection equation:

Translate constraints on Translate constraints on K K through projection equationthrough projection equation to to constraints on constraints on **

KRd0d]C

~|KR[IPXx

mapping between ∞ to an image is given by the planar homogaphy x=Hd, with H=KR

image of the absolute conic (IAC) = I

1-T-1T KKKKω 1TCHHC

(i) IAC depends only on intrinsics(ii) angle between two rays(iii) DIAC=*=KKT

(iv) K (cholesky factorisation)(v) image of circular points

2T

21T

1

2T

1

ωxxωxx

ωxxcos

Image of the absolute conic

HZ 7.5.1:

Constraints on *

1

ω 22

222

*

yx

yyyyxy

xyxyxx

cc

ccfccsf

cccsfcsf

Zero skewZero skew quadraticquadratic mm

Principal pointPrincipal point linearlinear 22mm

Zero skew (& p.p.)Zero skew (& p.p.) linearlinear mm

Fixed aspect ratio (& Fixed aspect ratio (& p.p.& Skew)p.p.& Skew)

quadraticquadratic m-1m-1

Known aspect ratio Known aspect ratio (& p.p.& Skew)(& p.p.& Skew)

linearlinear mm

Focal length Focal length

(& p.p. & Skew)(& p.p. & Skew)

linearlinear mm

*23

*13

*33

*12 ωωωω

0ωω *23

*13

0ω*12

*11

*22

*22

*11 ω'ωω'ω

*22

*11 ωω

*11

*33 ωω

condition constraint type #constraints

Summary: Self calibration based on the IADC

•Calibrated cameraCalibrated camera– Dual absolute quadric (DAC)Dual absolute quadric (DAC)– Dual image of the absolute conic (DIAC)Dual image of the absolute conic (DIAC)

•Projective cameraProjective camera– DACDAC– DIACDIAC

•AutocalibrationAutocalibration– Determine based on constraints onDetermine based on constraints on– Decompose Decompose

TKK

diagI

*

)0,1,1,1(~

Tii

iTii

T

KKPQP

HIHQ

**

* ~

* i*THIHQ

~*

i

j

Ci

Cj

Illustration of self-calibration

Projective Affine Metric

Degenerate configurations

• Pure translation: Pure translation: affine transformation (5 DOF)affine transformation (5 DOF)

• Pure rotation: Pure rotation: arbitrary pose for (3 DOF) arbitrary pose for (3 DOF)

• Planar motion: Planar motion: scaling axis perpendicular to plane scaling axis perpendicular to plane (1DOF)(1DOF)

• Orbital motion: Orbital motion: projective distortion along rotation axis projective distortion along rotation axis (2DOF) (2DOF)

π

Not unique solution !Not unique solution !

A complete modeling systemprojective

Sequence of frames scene structureSequence of frames scene structure

1.1. Get corresponding points (tracking).Get corresponding points (tracking).

2.2. 2,3 view geometry: 2,3 view geometry: compute F,T between consecutive frames compute F,T between consecutive frames (recompute correspondences).(recompute correspondences).

3.3. Initial reconstruction: Initial reconstruction: get an initial structure from a get an initial structure from a subsequence with big baseline (trilinear tensor, factorization …) subsequence with big baseline (trilinear tensor, factorization …) and bind more frames/points using resection/intersection.and bind more frames/points using resection/intersection.

4.4. Self-calibration.Self-calibration.

5.5. Bundle adjustment.Bundle adjustment.

A complete modeling systemaffine

Sequence of frames scene structureSequence of frames scene structure

1.1. Get corresponding points (tracking).Get corresponding points (tracking).

2.2. Affine factorization. (This already computes ML Affine factorization. (This already computes ML estimate over all frames so no need for bundle estimate over all frames so no need for bundle adjustment for simple scenes.adjustment for simple scenes.

3.3. Self-calibration.Self-calibration.

4.4. If several model segments: Merge, bundle adjust.If several model segments: Merge, bundle adjust.

Examples – modeling with dynamic texture

Cobzas,Yerex,JagersandCobzas,Yerex,Jagersand

Examples: geometric modeling

Debevec and Taylor:Debevec and Taylor: Façade Façade

Examples: geometric modeling

Pollefeys: Pollefeys: Arenberg CastleArenberg Castle

Examples: geometric modeling

INRIA –INRIA –VISIRE projectVISIRE project

Examples: geometric modeling

CIP Prague –CIP Prague –Projective Reconstruction Based on Cake ConfigurationProjective Reconstruction Based on Cake Configuration

Reading: FP Chapter 11.

• The Stereopsis Problem: Fusion and Reconstruction• Human Stereopsis and Random Dot Stereograms• Cooperative Algorithms• Correlation-Based Fusion• Multi-Scale Edge Matching• Dynamic Programming• Using Three or More Cameras

Dense stereo

•Go back to original images, do dense matching.Go back to original images, do dense matching.

•Try to get dense depth mapsTry to get dense depth maps

Stereo image rectification

Stereo image rectification

• reproject image planes onto a commonreproject image planes onto a common• plane parallel to the line between optical centersplane parallel to the line between optical centers• pixel motion is horizontal after this transformationpixel motion is horizontal after this transformation• two homographies (3x3 transform), one for each input two homographies (3x3 transform), one for each input

image reprojectionimage reprojectionC. Loop and Z. Zhang. C. Loop and Z. Zhang. Computing RectifyingComputing Rectifying Homographies Homographies

for Stereo Vision for Stereo Vision. IEEE Conf. Computer Vision and Pattern . IEEE Conf. Computer Vision and Pattern Recognition, 1999Recognition, 1999..

All epipolar lines are parallel in the rectified image plane.

Rectification

simplify stereo matching by warping the images

Apply projective transformation so that epipolar linescorrespond to horizontal scanlines

e

e

map epipole e to (1,0,0)

try to minimize image distortion

problem when epipole in (or close to) the image

He

0

0

1

Image rectification throughhomography warp

ExampleUnrectified

Rectified

Depth from disparity

f

x x’

baseline

z

C C’

X

f

Stereo matching algorithms•Match Pixels in Conjugate Epipolar LinesMatch Pixels in Conjugate Epipolar Lines

– Assume brightness constancyAssume brightness constancy– This is a tough problemThis is a tough problem– Numerous approachesNumerous approaches

– A good survey and evaluation: A good survey and evaluation: http://www.http://www.middleburymiddlebury..eduedu/stereo//stereo/

Your basic stereo algorithm

For each epipolar line

For each pixel in the left image• compare with every pixel on same epipolar line in right image

• pick pixel with minimum match cost

Improvement: match windows• This should look familar...

Stereo as energy minimization

• Find disparities d that minimize an energy function

• Simple pixel / window matching

SSD distance between windows I(x, y) and J(x, y + d(x,y))=

Stereo as energy minimization

I(x, y) J(x, y)

y = 141

C(x, y, d); the disparity space image (DSI)x

d

Stereo as energy minimization

y = 141

x

d

Simple pixel / window matching: choose the minimum of each column in the DSI independently:

Matching windowsSimilarity MeasureSimilarity Measure FormulaFormula

Sum of Absolute Differences Sum of Absolute Differences (SAD)(SAD)

Sum of Squared Differences (SSD)Sum of Squared Differences (SSD)

Zero-mean SADZero-mean SAD

Locally scaled SADLocally scaled SAD

Normalized Cross Correlation Normalized Cross Correlation (NCC)(NCC)

http://siddhantahuja.wordpress.com/category/stereo-vision/ SAD SSD NCC Ground truth

Stereo matching

Optimal path(dynamic programming )

Similarity measure(SSD or NCC)

Constraints• epipolar

• ordering

• uniqueness

• disparity limit

• disparity gradient limit

Trade-off

• Matching cost (data)

• Discontinuities (prior)

(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

Disparity map

image I(x,y) image I´(x´,y´)Disparity map D(x,y)

(x´,y´)=(x+D(x,y),y)

Hierarchical stereo matchingD

ow

nsam

plin

g

(Gau

ssia

n p

yra

mid

)

Dis

pari

ty p

rop

ag

ati

on

Allows faster computation

Deals with large disparity ranges

(Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

Example: reconstruct image from neighboring images