• date post

03-Jun-2020
• Category

## Documents

• view

0

0

Embed Size (px)

### Transcript of Lecture 10: Multi-view geometry - Artificial Lecture 10 - Fei-Fei Li 29 25-Oct-12 Dynamic...

• Lecture 10 -

Fei-Fei Li

Lecture 10: Multi-view geometry

Professor Fei-Fei Li Stanford Vision Lab

25-Oct-12 1

• Lecture 10 -

Fei-Fei Li

What we will learn today?

• Review for stereo vision • Correspondence problem (Problem Set 2 (Q3)) • Active stereo vision systems • Structure from motion

25-Oct-12 2

Reading: [HZ] Chapters: 4, 9, 11 [FP] Chapters: 10

• Lecture 10 -

Fei-Fei Li 25-Oct-12 3

Recovering structure from a single view Pinhole perspective projection

C

Ow

P p

Calibration rig

Scene

Camera K

Why is it so difficult?

Intrinsic ambiguity of the mapping from 3D to image (2D)

• Lecture 10 -

Fei-Fei Li 25-Oct-12 4

Two eyes help!

O2 O1

p2 p1

?

This is called triangulation

K2 =known K1 =known

R, T

• Lecture 10 -

Fei-Fei Li 25-Oct-12 5

Epipolar geometry

• Epipolar Plane • Epipoles e1, e2

• Epipolar Lines • Baseline

O1 O2

p2

P

p1

e1 e2

= intersections of baseline with image planes = projections of the other camera center = vanishing points of camera motion direction

• Lecture 10 -

Fei-Fei Li 25-Oct-12 6

Triangulation

O O’

p p’

P

0pFpT =′

F = Fundamental Matrix (Faugeras and Luong, 1992)

• Lecture 10 -

Fei-Fei Li 25-Oct-12 7

Why is F useful?

- Suppose F is known - No additional information about the scene and camera is given - Given a point on left image, how can I find the corresponding point on right image?

l’ = FT p p

• Lecture 10 -

Fei-Fei Li 25-Oct-12 8

Estimating F

O O’

p p’

P The Eight-Point Algorithm

(Hartley, 1995)

(Longuet-Higgins, 1981)

Lsq. solution by SVD!

• Rank 8 A non-zero solution exists (unique)

• Homogeneous system

• If N>8 F̂

f

1=f

W 0=f

W

• Lecture 10 -

Fei-Fei Li 25-Oct-12 9

Rectification

p’

P

p

O O’ • Make two camera images “parallel”

• Correspondence problem becomes easier

• Lecture 10 -

Fei-Fei Li 25-Oct-12 10

Application: view morphing S. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30

• Lecture 10 -

Fei-Fei Li

What we will learn today?

• Review for stereo vision • Correspondence problem (Problem Set 2 (Q3)) • Active stereo vision systems • Structure from motion

25-Oct-12 11

Reading: [HZ] Chapters: 4, 9, 11 [FP] Chapters: 10

• Lecture 10 -

Fei-Fei Li 25-Oct-12 12

Stereo-view geometry

• Correspondence: Given a point in one image, how

can I find the corresponding point p’ in another one?

• Camera geometry: Given corresponding points in two images, find camera matrices, position and pose.

• Scene geometry: Find coordinates of 3D point from its projection into 2 or multiple images.

Previous lecture (#9)

• Lecture 10 -

Fei-Fei Li 25-Oct-12 13

Stereo-view geometry

• Correspondence: Given a point in one image, how

can I find the corresponding point p’ in another one?

• Camera geometry: Given corresponding points in two images, find camera matrices, position and pose.

• Scene geometry: Find coordinates of 3D point from its projection into 2 or multiple images.

This lecture (#10)

• Lecture 10 -

Fei-Fei Li 25-Oct-12 14

Depth estimation Pinhole perspective projection

Subgoals: - Solve the correspondence problem - Use corresponding observations to triangulate

p’

P

p

O1 O2

• Lecture 10 -

Fei-Fei Li 25-Oct-12 15

Correspondence problem Pinhole perspective projection Pinhole perspective projection

p’

P

p

O1 O2

Given a point in 3D, discover corresponding observations in left and right images [also called binocular fusion problem]

• Lecture 10 -

Fei-Fei Li 25-Oct-12 16

Correspondence problem Pinhole perspective projection Pinhole perspective projection

•A Cooperative Model (Marr and Poggio, 1976)

•Correlation Methods (1970--)

•Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)

[FP] Chapters: 11

• Lecture 10 -

Fei-Fei Li 25-Oct-12 17

Correlation Methods Pinhole perspective projection

p

• Pick up a window around p(u,v)

• Lecture 10 -

Fei-Fei Li 25-Oct-12 18

Correlation Methods Pinhole perspective projection

• Pick up a window around p(u,v) • Build vector W • Slide the window along v line in image 2 and compute w’ • Keep sliding until w∙w’ is maximized.

• Lecture 10 -

Fei-Fei Li 25-Oct-12 19

Correlation Methods Pinhole perspective projection

Normalized Correlation; maximize: )ww)(ww( )ww)(ww( ′−′− ′−′−

• Lecture 10 -

Fei-Fei Li 25-Oct-12 20

Correlation Methods Pinhole perspective projection Left Right

scanline

Norm. corr Credit slide S. Lazebnik

• Lecture 10 -

Fei-Fei Li 25-Oct-12 21

Correlation Methods Pinhole perspective projection

– Smaller window - More detail - More noise

– Larger window

- Smoother disparity maps - Less prone to noise

Window size = 3 Window size = 20

Credit slide S. Lazebnik

• Lecture 10 -

Fei-Fei Li 25-Oct-12 22

Issues with Correlation Methods Pinhole perspective projection

•Fore shortening effect

•Occlusions

O O’

O O’

It is desirable to have small B/z ratio!

• Lecture 10 -

Fei-Fei Li 25-Oct-12 24

Issues with Correlation Methods Pinhole perspective projection

•Homogeneous regions

Hard to match pixels in these regions

• Lecture 10 -

Fei-Fei Li 25-Oct-12 25

Issues with Correlation Methods Pinhole perspective projection

•Repetitive patterns

• Lecture 10 -

Fei-Fei Li 25-Oct-12 26

Results with window search Pinhole perspective projection Data

Ground truth Window-based matching

Credit slide S. Lazebnik

• Lecture 10 -

Fei-Fei Li 25-Oct-12 27

Improving correspondence: Non-local constraints

• Uniqueness – For any point in one image, there should be at most one

matching point in the other image

• Lecture 10 -

Fei-Fei Li 25-Oct-12 28

Improving correspondence: Non-local constraints

• Uniqueness – For any point in one image, there should be at most one

matching point in the other image

• Ordering – Corresponding points should be in the same order in both

views

Courtesy J. Ponce

• Lecture 10 -

Fei-Fei Li 25-Oct-12 29

Dynamic Programming Pinhole perspective projection (Baker and Binford, 1981)

•Nodes = matched feature points (e.g., edge points). • Arcs = matched intervals along the epipolar lines. • Arc cost = discrepancy between intervals. Find the minimum-cost path going monotonically down and right from the top-left corner of the graph to its bottom-right corner.

[Uses ordering constraint]

Courtesy J. Ponce

• Lecture 10 -

Fei-Fei Li 25-Oct-12 30

Dynamic Programming (Baker and Binford, 1981)

(Ohta and Kanade, 1985)

• Lecture 10 -

Fei-Fei Li 25-Oct-12 31

Improving correspondence: Non-local constraints

• Uniqueness – For any point in one image, there should be at most one

matching point in the other image

• Ordering – Corresponding points should be in the same order in both

views

• Smoothness Disparity is typically a smooth function of x (except in occluding boundaries)

• Lecture 10 -

Fei-Fei Li 25-Oct-12 32

Stereo matching as energy minimization Pinhole perspective projection I1 I2 D

• Energy functions of this form can be minimized using graph cuts

Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 01

W1(i ) W2(i+D(i )) D(i )

)(),,( smooth21data DEDIIEE βα +=

( )∑ −= ji

jDiDE ,neighbors

smooth )()(ρ( )221data ))(()(∑ +−= i

iDiWiWE

• Lecture 10 -

Fei-Fei Li 25-Oct-12 33

Stereo matching as energy minimization Pinhole perspective projection