EECS 442 – Computer vision Stereo...

EECS 442 Computer vision

Stereo systems

Stereo visionRectificationCorrespondence problemActive stereo vision systems

Reading: [HZ] Chapter: 11[FP] Chapter: 11

Stereo vision

Goal: estimate the position of P given the observation ofP from two view pointsAssumptions: known camera parameters and position (K, R, T)

p

P

p

O1 O2

Stereo vision

Subgoals: - Solve the correspondence problem- Use corresponding observations to triangulate

p

P

p

O1 O2

Correspondence problem

p

P

p

O1 O2

Given a point in 3d, discover corresponding observations in left and right images [also called binocular fusion problem]

Triangulation

Intersecting the two lines of sight gives rise to P

p

P

p

O1 O2

Epipolar Plane Epipoles e1, e2

Epipolar Lines

Baseline

Epipolar geometry

O1 O2

p

P

p

e1 e2

= intersections of baseline with image planes = projections of the other camera center= vanishing points of camera motion direction

O O

P

e2p pe2

Parallel image planes

K Kx

z

v = v

y

Rectification: making two images parallel

u

v

u

v


== 00

00000

][T

TRtEK1=K2 = known

x parallel to O1O2

O O

P

e2p pe2

K Kx

z

y u

v

u

v

v = v?


( ) ( ) vTTvvTTvuv

u

TTvu ==

=

0

010

10000

0001

O O

P

e2p pe2

K Kx

z

y u

v

u

v

0pEpT =

Making image planes parallel

p

P

p

O O

H

GOAL: Estimate the perspective transformation Hthat makes the images parallel

Projective transformation

ix ix

H

ii xHx =

Now we dont have the destination image


p

P

p

O O

H

GOAL: Estimate the perspective transformation Hthat makes the images parallel

Impose v=v How? Map e to infinity


T21

T ]1ee[TRKe == TKe =


p

P

p

e eK K

R,T

[ ] PTRKP [ ] P0IKP

O O


1. Map e to the x-axis at location [1,0,1]T

[ ]T101HH1 TRH = = T21 ]1ee[e

p

P

p

e eK K

R,T

[ ] P0IKP

O O

[ ] PTRKP


2. Send epipole to infinity

pp

[ ]T101e =

[ ]T001e =

=

101010001

H2

P

[ ]T001

e

O O

Minimizes the distortion in a neighborhood (approximates id. mapping)


4. Align epipolar lines

pp

H = H1 H2

P

e

O O

e

3. Define: H = H1 H2

Note: H is not unique!

Projective transformation of a line (in 2D)

lHl T=

=

bvtA

H


pp

H = H1 H2

P

e

O O

e

4. Align epipolar lines

3. Define: H = H1 H2

lHlH TT =

ll

H, H called matched pair of transformation

[HZ] Chapters: 11 (sec. 11.12)


H

Courtesy figure S. Lazebnik

Why rectification is useful?

Makes the correspondence problem easier Makes triangulation easy

Computing depth

O O

P

e2p pe2

K Kx

z

y u

v

u

v

f

x x

Baseline B

z

O O

P

f

zfBxx =

Note: Disparity is inversely proportional to depth

Computing depth

= disparity

Disparity maps

http://vision.middlebury.edu/stereo/Disparity map / depth map

Stereo pair

Disparity map with occlusions

x x

zfBxx =

Stereo systems



p

P

p

O1 O2

Given a point in 3d, discover corresponding observations in left and right images [also called binocular fusion problem]

A Cooperative Model (Marr and Poggio, 1976)

Correlation Methods (1970--)

Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)


[FP] Chapters: 11


Slide the window along the epipolar line until ww is maximized.

courtesy slide to J. Ponce


Normalized Correlation; minimize:)ww)(ww()ww)(ww(

Left Right

scanline

Correlation methods

Norm. corrCredit slide S. Lazebnik

Smaller window- More detail- More noise

Larger window- Smoother disparity maps- Less prone to noise

W = 3 W = 20

Correlation methods

Credit slide S. Lazebnik

Issues

Fore shortening effect

Occlusions

OO

O O

It is desirable to have small B/z ratio!

O O

Issues

O O

small B/z ratio

Issues

Small error in measurements implies large error in estimating depth

Homogeneous regions

Issues

Hard to match pixels in these regions

Repetitive patterns

Issues

Correspondence problem is difficult!

- Occlusions- Fore shortening- Baseline trade-off- Homogeneous regions- Repetitive patterns

Apply non-local constraints to help enforce the correspondences

Results with window search

Data

Ground truth


Window-based matching

Improving correspondence: Non-local constraints

Uniqueness For any point in one image, there should be at most one

matching point in the other image




Ordering Corresponding points should be in the same order in both

views






views

Not alwaysin presenceof occlusions!



Dynamic Programming (Baker and Binford, 1981)

Nodes = matched feature points (e.g., edge points). Arcs = matched intervals along the epipolar lines. Arc cost = discrepancy between intervals.

Find the minimum-cost path going monotonicallydown and right from the top-left corner of thegraph to its bottom-right corner.

[Uses ordering constraint]


Dynamic Programming (Baker and Binford, 1981)


Dynamic Programming (Ohta and Kanade, 1985)

Reprinted from Stereo by Intra- and Intet-Scanline Search, by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and MachineIntelligence, 7(2):139-154 (1985). 1985 IEEE.




views

Smoothness Disparity is typically a smooth function of x (expect in occluding

boundaries)


Smoothness

Stereo matching as energy minimization

I1

W1(i)

I2

W2(i+D(i))

D

D(i)

Energy functions of this form can be minimized using graph cuts

Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 01

)(),,( smooth21data DEDIIEE +=

( ) =ji

jDiDE,neighbors

smooth )()(( )221data ))(()( +=i

iDiWiWE


Graph cutsGround truth Window-based matching

Stereo matching as energy minimization

Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 01

http://www.middlebury.edu/stereo/

Two-frame stereo correspondence algorithms

Click here

http://www.middlebury.edu/stereo/

stereo vision software development kit.Stereo SDKA. Criminisi, A. Blake and D. Robertson

Application foreground/background separationV. Kolmogorov, A. Criminisi, A. Blake, G. Cross and C. Rother. Bi-layer segmentation of binocular stereo video CVPR 2005

http://research.microsoft.com/~antcrim/demos/ACriminisi_Recognition_CowDemo.wmv

Click on ACriminisi_i2i_BilayerSegmentation.wmv

http://research.microsoft.com/users/antcrim/papers/Criminisi_cvpr2005.pdf

Application 3D Urban Scene Modeling3D Urban Scene Modeling Integrating Recognition and Reconstruction,N. Cornelis, B. Leibe, K. Cornelis, L. Van Gool, IJCV 08.

http://www.vision.ee.ethz.ch/showroom/index.en.html#

Click on cornelis-cognitiveloops-vpcvpr06.avi

Stereo systems


Active stereo (point)

Replace one of the two cameras by a projector- Single camera - Projector geometry calibrated- Correspondence problem solved!

P

projector O2

p

Active stereo (stripe)

projectorO

-Projector and camera are parallel- Correspondence problem solved!

Laser scanning

Optical triangulation Project a single stripe of laser light Scan it across the surface of the object This is a very precise version of structured light scanning

Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/

Source: S. Seitz

The Digital Michelangelo Project, Levoy et al.Source: S. Seitz

Laser scanning

Light source O

Active stereo (shadows)J. Bouguet & P. Perona, 99

- 1 camera,1 light source- very cheap setup- calibrated the light source

Active stereo (shadows)

Click here

Active stereo (dense)

- Dense reconstruction - Correspondence problem again- Get around it by using color codes

projectorO

L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002

L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002

Rapid shape acquisition: Projector + stereo cameras

Next lecture

Volumetric stereo

Human Stereopsis

Figure from US Navy Manual of Basic Optics and Optical Instruments, prepared by Bureau of Naval Personnel. Reprinted by Dover Publications, Inc., 1969.

Credit slide J. Ponce

Human Stereopsis: Reconstruction

Disparity: d = r-l = D-F;

d=0

d

What if F is not known?

Helmoltz (1909):

There is evidence showing the vergence anglescannot be measured precisely.

Humans get fooled by bas-relief sculptures.

Human Stereopsis: Reconstruction


Human Stereopsis: Binocular Fusion


How are the correspondences established?

Julesz (1971): Is the mechanism for binocular fusiona monocular process or a binocular one?? There is anecdotal evidence for the latter (camouflage).

Random dot stereograms provide an objective answer

Human Stereopsis: Binocular Fusion


EECS 442 Computer visionStereo systems Stereo systems Correlation methodsCorrelation methodsResults with window searchImproving correspondence: Non-local constraintsImproving correspondence: Non-local constraintsImproving correspondence: Non-local constraintsImproving correspondence: Non-local constraintsSmoothnessStereo matching as energy minimizationStereo matching as energy minimizationStereo systems Laser scanningLaser scanningNext lecture

EECS 442 – Computer vision Stereo...

Documents

Transcript of EECS 442 – Computer vision Stereo...