Computing F and rectification

class 14

Multiple View GeometryComp 290-089Marc Pollefeys

Multiple View Geometry course schedule(subject to change)

Jan. 7, 9 Intro & motivation Projective 2D Geometry

Jan. 14, 16

(no class) Projective 2D Geometry

Jan. 21, 23

Projective 3D Geometry (no class)

Jan. 28, 30

Parameter Estimation Parameter Estimation

Feb. 4, 6 Algorithm Evaluation Camera Models

Feb. 11, 13

Camera Calibration Single View Geometry

Feb. 18, 20

Epipolar Geometry 3D reconstruction

Feb. 25, 27

Fund. Matrix Comp. Structure Comp.

Mar. 4, 6 Planes & Homographies Trifocal Tensor

Mar. 18, 20

Three View Reconstruction

Multiple View Geometry

Mar. 25, 27

MultipleView Reconstruction

Bundle adjustment

Apr. 1, 3 Auto-Calibration Papers

Apr. 8, 10

Dynamic SfM Papers

Apr. 15, 17

Cheirality Papers

Apr. 22, 24

Duality Project Demos

Two-view geometry

Epipolar geometry

3D reconstruction

F-matrix comp.

Structure comp.

Epipolar geometry: direct computation

0Fxx'T

0f1''''''

1'''''' 111111111111

nnnnnnnnnnnn yxyyyxyxyxxx

yxyyyxyxyxxx

0)λIFFdet(Fdet)λFFdet( 1-12221

Basic equation

8-point algorithm (normalize!)

7-point algorithm (impose rank 2)

Epipolar geometry: iterative computation

Maximum Likelihood Estimation

i

iiii dd 22 'x̂,x'x̂,x

(= least-squares for Gaussian noise)

0x̂F'x̂ subject to T

22

21

2

2T2

1T

2T

FxFxFx'Fx'

Fxx'

2

221

2

2T2

1T

T

FxFx

1

Fx'Fx'

1Fxx'

i

iiii dFd2T2 x'F,xx,x'

Sampson error (first order approx. to MLE)

Symmetric epipolar error

Automatic computation of F

(i) Interest points(ii) Putative correspondences(iii) RANSAC (iv) Non-linear re-estimation of F(v) Guided matching(repeat (iv) and (v) until stable)

• Extract feature points to relate images

• Required properties:• Well-defined

(i.e. neigboring points should all be different)

• Stable across views(i.e. same 3D point should be extracted as feature for neighboring viewpoints)

Feature points

homogeneous

edge

corner

dxdyyxwyI

xI

W yIxI

,

M

M should have large eigenvalues

(e.g.Harris&Stephens´88; Shi&Tomasi´94)

MTSSD

Find points that differ as much as possible from all neighboring points

Feature = local maxima (subpixel) of F(1, 2)

Feature points

Select strongest features (e.g. 1000/image)

Feature points

Evaluate NCC for all features withsimilar coordinates

Keep mutual best matchesKeep mutual best matches

Still many wrong matches!Still many wrong matches!

10101010 ,,´´, e.g. hhww yyxxyx

?

Feature matching

0.96 -0.40 -0.16 -0.39 0.19

-0.05 0.75 -0.47 0.51 0.72

-0.18 -0.39 0.73 0.15 -0.75

-0.27 0.49 0.16 0.79 0.21

0.08 0.50 -0.45 0.28 0.99

1 5

24

3

1 5

24

3

Gives satisfying results for small image motions

Feature example

• Requirement to cope with larger variations between images• Translation, rotation, scaling• Foreshortening• Non-diffuse reflections• Illumination

geometric transformations

photometric changes

Wide-baseline matching…

Wide baseline matching for two different region types

(Tuytelaars and Van Gool BMVC 2000)

Wide-baseline matching…

Step 1. Extract featuresStep 2. Compute a set of potential matchesStep 3. do

Step 3.1 select minimal sample (i.e. 7 matches)

Step 3.2 compute solution(s) for F

Step 3.3 determine inliers

until (#inliers,#samples)<95%

samples#7)1(1

matches#inliers#

#inliers 90%

80%

70% 60%

50%

#samples

5 13 35 106 382

Step 4. Compute F based on all inliersStep 5. Look for additional matchesStep 6. Refine F based on all correct matches

(generate hypothesis)

(verify hypothesis)

RANSAC

restrict search range to neighborhood of epipolar line (1.5 pixels)

relax disparity restriction (along epipolar line)

Finding more matches

• Degenerate cases• Planar scene• Pure rotation

• No unique solution• Remaining DOF filled by noise• Use simpler model (e.g. homography)

• Model selection (Torr et al., ICCV´98, Kanatani, Akaike)

• Compare H and F according to expected residual error (compensate for model complexity)

Degenerate cases:

Model selection

n = number of measurements (inliers+outliers)r = dimension of datak = motion model parametersd = dimension of structure

structure

motion

MLE

Geometric Robust Information Criterion

Model selection

Video tracking

H

F

Dominant planes

• Absence of sufficient features (no texture)• Repeated structure ambiguity

(Schaffalitzky and Zisserman, BMVC‘98)

• Robust matcher also finds Robust matcher also finds support for wrong hypothesissupport for wrong hypothesis• solution: detect repetition solution: detect repetition

More problems:

geometric relations between two views is fully

described by recovered 3x3 matrix F

two-view geometry

Image pair rectification

simplify stereo matching by warping the images

Apply projective transformation so that epipolar linescorrespond to horizontal scanlines

e

e

map epipole e to (1,0,0)

try to minimize image distortion

problem when epipole in (or close to) the image

Planar rectification

Bring two views Bring two views to standard stereo setupto standard stereo setup

(moves epipole to )(not possible when in/close to image)

~ image size

(calibrated)(calibrated)

Distortion minimization(uncalibrated)

(standard approach)

Polar re-parameterization around epipolesRequires only (oriented) epipolar geometryPreserve length of epipolar linesChoose so that no pixels are compressed

original image rectified image

Polar rectification(Pollefeys et al. ICCV’99)

Works for all relative motionsGuarantees minimal image size

polar rectification: example

polar rectification: example

Example: Béguinage of Leuven

Does not work with standard Homography-based approaches

Example: Béguinage of Leuven

Exploiting motion and scene constraints

• Ordering constraint• Uniqueness constraint• Disparity limit• Disparity continuity constraint

• Epipolar constraint Epipolar constraint (through rectification)

Ordering constraint

11 22 33 4,54,5 66 11 2,32,3 44 55 66

2211 33 4,54,5 6611

2,32,3

44

55

66

surface slicesurface slice surface as a pathsurface as a path

occlusion right

occlusion left

Uniqueness constraint

• In an image pair each pixel has at most one corresponding pixel• In general one corresponding pixel• In case of occlusion there is none

Disparity constraint

surface slicesurface slice surface as a pathsurface as a path

bounding box

dispa

rity b

and

use reconsructed features to determine bounding box

constantdisparitysurfaces

Disparity continuity constraint

• Assume piecewise continuous surface

piecewise continuous disparity• In general disparity changes

continuously• discontinuities at occluding

boundaries

Stereo matching

Optimal path(dynamic programming )

Similarity measure(SSD or NCC)

Constraints• epipolar

• ordering

• uniqueness

• disparity limit

• disparity gradient limit

Trade-off

• Matching cost (data)

• Discontinuities (prior)

(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

Hierarchical stereo matching

Dow

nsam

plin

g

(Gau

ssia

n p

yra

mid

)

Dis

pari

ty p

rop

ag

ati

on

Allows faster computation

Deals with large disparity ranges

(Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

Disparity map

image I(x,y) image I´(x´,y´)Disparity map D(x,y)

(x´,y´)=(x+D(x,y),y)

Example: reconstruct image from neighboring

images

Multi-view depth fusion

• Compute depth for every pixel of reference image• Triangulation• Use multiple views• Up- and down

sequence• Use Kalman filter

(Koch, Pollefeys and Van Gool. ECCV‘98)

Allows to compute robust texture

Assignment 2Compute F automatically from image

pair

(matches, 7-point, RANSAC, 8-point, iterative,more matches, epipolar lines, etc.)

(due by Wednesday 19/03/03)

Next class: reconstructing points and lines