Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there...
Transcript of Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there...
![Page 1: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/1.jpg)
Stereo Vision (Correspondences)
Instructor: Jason Corso (jjcorso)!web.eecs.umich.edu/~jjcorso/t/598F14!
!
Materials on these slides have come from many sources in addition to myself; individual slides reference specific sources. The primary source for these slides is Silvio Savarese.!
EECS 598-08 Fall 2014!Foundations of Computer Vision!!
Readings: FP 7; SZ 11; TV 7!Date: 10/27/14!!
![Page 2: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/2.jpg)
Plan
• Stereo vision!• Rectification!• Correspondence problem!• Active stereo vision systems!
2
![Page 3: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/3.jpg)
Goal: estimate the position of P given the observation of!P from two view points!Assumptions: known camera parameters and position (K, R, T)!
p’!
P!
p!
O1! O2!
Stereo Vision
Slide source: S. Savarese.!
![Page 4: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/4.jpg)
Subgoals: !- Solve the correspondence problem!- Use corresponding observations to triangulate!
p’!
P!
p!
O1! O2!
Stereo Vision
Slide source: S. Savarese.!
![Page 5: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/5.jpg)
p’!
P!
p!
O1! O2!
• Given a point in 3d, discover corresponding observations in left and right images!
The Correspondence Problem
Slide source: S. Savarese.!
![Page 6: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/6.jpg)
• Intersecting the two lines of sight gives rise to P!
p’!
P!
p!
O1! O2!
Triangulation
Slide source: S. Savarese.!
![Page 7: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/7.jpg)
• Epipolar Plane! • Epipoles e1, e2!
• Epipolar Lines!• Baseline!
O1! O2!
p’!
P!
p!
e1! e2!
= intersections of baseline with image planes != projections of the other camera center!!
Epipolar Geometry
Slide source: S. Savarese.!
![Page 8: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/8.jpg)
The Epipolar Constraint
• E p2 is the epipolar line associated with p2 (l1 = E p2)!• ET p1 is the epipolar line associated with p1 (l2 = ET p1)!• E e2 = 0 and ET e1 = 0!• E is 3x3 matrix; 5 DOF!• E is singular (rank two)! !
O1! O2!
p2!
P!
p1!
e1!e2!
l1! l2!
021 =⋅ pEp T
![Page 9: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/9.jpg)
• When views are parallel both correspondence and triangulation become much easier!!
p’!
P!
p!
O! O’!
Parallel Image Planes
Slide source: S. Savarese.!
![Page 10: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/10.jpg)
O! O!
P!
e2!p! p’!
e2!
K! K!x!
y!
z!
Rectification: making two images “parallel” !
• Parallel epipolar lines!• Epipoles at infinity !• v = v’!
u!
v!
u!
v!
Parallel Image Planes
Slide source: S. Savarese.!
![Page 11: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/11.jpg)
K1=K2 = known!x parallel to O1O2!
RtE ][ ×=
O! O!
P!
e2!p! p’!
e2!
K! K!x!
y!
z!
u!
v!
u!
v!
Parallel Image Planes
Slide source: S. Savarese.!
![Page 12: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/12.jpg)
baba ][0
00
×=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−
−
−
=×
z
y
x
xy
xz
yz
bbb
aaaaaa
Cross product as matrix multiplication 12
Slide source: S. Savarese.!
![Page 13: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/13.jpg)
K1=K2 = known!x parallel to O1O2!
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−== ×
0000
000][
TTRtE
O! O!
P!
e2!p! p’!
e2!
K! K!x!
y!
z!
u!
v!
u!
v!
→ v = v’?!
Parallel Image Planes
Slide source: S. Savarese.!
![Page 14: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/14.jpg)
( ) ( ) vTTvvTTvuv
u
TTvu ʹ′==
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
ʹ′
−=⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛ʹ′
ʹ′
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
− 00
10100
00000
1
O! O!
P!
e2!p! p’!
e2!
K! K!x!
y!
z!
u!
v!
u!
v!
0pEpT =ʹ′
Parallel Image Planes
Slide source: S. Savarese.!
![Page 15: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/15.jpg)
p’!
P!
p!
O! O’!
H!
GOAL: Estimate the perspective transformation H!! that makes the images parallel!
Making Image Planes Parallel
Slide source: S. Savarese.!
![Page 16: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/16.jpg)
ix ixʹ′
H!
ii xHx =ʹ′
Now we don’t have the destination image L!Slide source: S. Savarese.!
Making the Image Planes Parallel The Projective Transformation
![Page 17: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/17.jpg)
p’!
P!
p!
O! O’!
H!
GOAL: Estimate the perspective transformation H that makes images parallel!
Impose v’=v!• This leaves degrees of freedom for determining H!• If an inappropriate H is chosen, severe projective distortions on image take place!• We impose a number of restrictions while computing H!
Slide source: S. Savarese.!
Making the Image Planes Parallel
![Page 18: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/18.jpg)
TRKeee TT == ]1[ 21 TKe ʹ′=ʹ′
p’!
P!
p!
e! e’!
R,T!
[ ] PTRKP ʹ′→[ ] P0IKP→
O! O’!
0. Compute epipoles!
Slide source: S. Savarese.!
Making the Image Planes Parallel
![Page 19: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/19.jpg)
1. Map e to the x-axis at location [1,0,1]T (normalization)!
[ ]T101
→= T21 ]1ee[e
p’!
P!
p!
e’!
O! O’!
HHTRH =1
e!
Slide source: S. Savarese.!
Making the Image Planes Parallel
![Page 20: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/20.jpg)
2. Send epipole to infinity:!
p’!p!
[ ]T101e =
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−
=
101010001
H2
→
P!
[ ]T001
e!
O! O’!
Minimizes the distortion in a !neighborhood (approximates id. mapping)!
Slide source: S. Savarese.!
Making the Image Planes Parallel
![Page 21: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/21.jpg)
4. Align epipolar lines!
p’!p!
H = H2 H1!
P!
e!
O! O’!
e’!
3. Define: H = H2 H1!
Slide source: S. Savarese.!
Making the Image Planes Parallel
![Page 22: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/22.jpg)
!l →H −T l
⎥⎦
⎤⎢⎣
⎡=
bvtA
H
Slide source: S. Savarese.!
Projective Transformation of a Line
![Page 23: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/23.jpg)
p’!p!
H = H2 H1!
P!
e!
O! O’!
e’!
4. Align epipolar lines!
3. Define: H = H2 H1! lHlH TT −− =ʹ′ʹ′These are called matched pair of transformation!
l lʹ′
[HZ] Chapters: 11 (sec. 11.12)!Slide source: S. Savarese.!
Making the Image Planes Parallel
![Page 24: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/24.jpg)
H!
Courtesy figure S. Lazebnik!Slide source: S. Savarese.!
Making the Image Planes Parallel
![Page 25: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/25.jpg)
• Makes the correspondence problem easier!• Makes triangulation easy!
Slide source: S. Savarese.!
Why is Rectification Useful?
![Page 26: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/26.jpg)
S. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30 !
Slide source: S. Savarese.!
Application: View Morphing
![Page 27: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/27.jpg)
Slide source: S. Savarese.!
Morphing Without Using Geometry
![Page 28: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/28.jpg)
Slide source: S. Savarese.!
Rectification
![Page 29: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/29.jpg)
Slide source: S. Savarese.!
![Page 30: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/30.jpg)
Subgoals: !- Solve the correspondence problem!- Use corresponding observations to triangulate!
p’!
P!
p!
O1! O2!
Slide source: S. Savarese.!
Stereo Vision
![Page 31: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/31.jpg)
O! O’!
P!
e2!p! p’!
e2!
K! K!x!
y!
z!
u!
v!
u!
v!
Slide source: S. Savarese.!
Computing Depth
![Page 32: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/32.jpg)
f
x x’
Baseline B!
z
O O’
P
f
Computing Depth
zfBxx ⋅
=ʹ′−
Note: Disparity is inversely proportional to depth!
= disparity!
Slide source: S. Savarese.!
![Page 33: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/33.jpg)
f!
Z!
Computing Depth 33
O2! O1!
x2!x1!c2! c1!
X
u2! u1!
T!
![Page 34: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/34.jpg)
http://vision.middlebury.edu/stereo/!
Stereo pair!
Disparity map / depth map! Disparity map with occlusions!
x! x’!
Slide source: S. Savarese.!
Disparity Maps
![Page 35: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/35.jpg)
Subgoals: !- Solve the correspondence problem!- Use corresponding observations to triangulate!
p’!
P!
p!
O1! O2!
Slide source: S. Savarese.!
Stereo Vision
![Page 36: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/36.jpg)
p’!
P!
p!
O1! O2!
Given a point in 3d, discover corresponding observations !in left and right images [also called binocular fusion problem]!
Slide source: S. Savarese.!
The Correspondence Problem
![Page 37: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/37.jpg)
• A Cooperative Model (Marr and Poggio, 1976)!
• Correlation Methods (1970--)!
• Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)!
! [FP] Chapters: 8 !!
Slide source: S. Savarese.!
The Correspondence Problem
![Page 38: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/38.jpg)
210!
p! p!
195! 150! 210!
Slide source: S. Savarese.!
Correlation Methods (1970--)
![Page 39: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/39.jpg)
• Pick up a window around p(u,v)!
p!
Slide source: S. Savarese.!
Correlation Methods (1970--)
![Page 40: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/40.jpg)
• Pick up a window around p(u,v)!• Build vector W!• Slide the window along v line in image 2 and compute w’!• Keep sliding until w∙w’ is maximized.!
!
Slide source: S. Savarese.!
Correlation Methods (1970--)
![Page 41: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/41.jpg)
Normalized Correlation; minimize:!)ww)(ww()ww)(ww(ʹ′−ʹ′−
ʹ′−ʹ′−
Slide source: S. Savarese.!
Correlation Methods (1970--)
![Page 42: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/42.jpg)
Left! Right!
scanline!
Norm. corr!Credit slide S. Lazebnik!
![Page 43: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/43.jpg)
– Smaller window!- More detail!- More noise !
– Larger window!- Smoother disparity maps!- Less prone to noise!
Window size = 3 Window size = 20
Credit slide S. Lazebnik!
Correlation Methods
![Page 44: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/44.jpg)
• Fore shortening effect!
• Occlusions!
O! O’!
O! O’!
Slide source: S. Savarese.!
Issues
![Page 45: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/45.jpg)
O! O’!
• Small error in measurements implies large error in estimating depth!
• It is desirable to have small T/z ratio!!
Slide source: S. Savarese.!
Issues
![Page 46: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/46.jpg)
• Homogeneous regions !
Hard to match pixels in these regions!
Slide source: S. Savarese.!
Issues
![Page 47: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/47.jpg)
• Repetitive patterns !
Slide source: S. Savarese.!
Issues
![Page 48: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/48.jpg)
- Occlusions!- Fore shortening!- Baseline trade-off!- Homogeneous regions!- Repetitive patterns!
Apply non-local constraints to help enforce the correspondences!
Slide source: S. Savarese.!
The Correspondence Problem is Difficult
![Page 49: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/49.jpg)
Results with window search Data!
Ground truth!
Credit slide S. Lazebnik!
Window-based matching!
![Page 50: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/50.jpg)
Improving correspondence: Non-local constraints
• Uniqueness !– For any point in one image, there should be at most one
matching point in the other image!
Slide source: S. Savarese.!
![Page 51: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/51.jpg)
• Uniqueness !– For any point in one image, there should be at most one matching point in the other image!
• Ordering!– Corresponding points should be in the same order in both views!
Not always!in presence!of occlusions!!
Improving correspondence: Non-local constraints
courtesy slide to J. Ponce!
![Page 52: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/52.jpg)
Dynamic Programming (Baker and Binford, 1981)!
• Nodes = matched feature points (e.g., edge points).!• Arcs = matched intervals along the epipolar lines.!• Arc cost = discrepancy between intervals.!!Find the minimum-cost path going monotonically!down and right from the top-left corner of the!graph to its bottom-right corner.!!
[Uses ordering constraint]!
courtesy slide to J. Ponce!
![Page 53: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/53.jpg)
Dynamic Programming (Baker and Binford, 1981)!
courtesy slide to J. Ponce!
![Page 54: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/54.jpg)
Dynamic Programming (Ohta and Kanade, 1985)!
Reprinted from “Stereo by Intra- and Intet-Scanline Search,” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and Machine!Intelligence, 7(2):139-154 (1985). © 1985 IEEE.!
![Page 55: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/55.jpg)
• Uniqueness !– For any point in one image, there should be at most one
matching point in the other image!
• Ordering!– Corresponding points should be in the same order in both views!
• Smoothness!– Disparity is typically a smooth function of x (expect in occluding
boundaries)!
Improving correspondence: Non-local constraints
Slide source: S. Savarese.!
![Page 56: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/56.jpg)
Slide source: S. Savarese.!
Smoothness
![Page 57: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/57.jpg)
I1 I2 D
• Energy functions of this form can be minimized using graph cuts!
W1(i ) W2(i+D(i )) D(i )
)(),,( smooth21data DEDIIEE βα +=
( )∑ −=ji
jDiDE,neighbors
smooth )()(ρ( )221data ))(()(∑ +−=i
iDiWiWE
Credit slide S. Lazebnik!
Stereo Matching as Energy Minimization Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 01!
![Page 58: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/58.jpg)
Graph cuts!Ground truth!
Stereo Matching as Energy Minimization
Window-based !matching!
Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 01!
Slide source: S. Savarese.!
![Page 59: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/59.jpg)
stereo vision software development kit. !Stereo SDK!A. Criminisi, A. Blake and D. Robertson!
![Page 60: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/60.jpg)
V. Kolmogorov, A. Criminisi, A. Blake, G. Cross and C. Rother. !Bi-layer segmentation of binocular stereo video CVPR 2005 !
http://research.microsoft.com/~antcrim/demos/ACriminisi_Recognition_CowDemo.wmv!
Application: Foreground/Background Segmentation
![Page 61: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/61.jpg)
3D Urban Scene Modeling Integrating Recognition and Reconstruction,!N. Cornelis, B. Leibe, K. Cornelis, L. Van Gool, IJCV 08. !
http://www.vision.ee.ethz.ch/showroom/index.en.html#!
Application: 3D Urban Scene Modeling
Link to movie.!
![Page 62: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/62.jpg)
Replace one of the two cameras by a projector!- Single camera !- Projector geometry calibrated!- What’s the advantage of having the projector? !
P!
projector! O2!
p’!
Correspondence problem solved!!Slide source: S. Savarese.!
Active Stereo: Point
![Page 63: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/63.jpg)
projector! O!
- Projector and camera are parallel!- Correspondence problem solved!!
Slide source: S. Savarese.!
Active Stereo: Stripe
![Page 64: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/64.jpg)
Laser scanning
• Optical triangulation!– Project a single stripe of laser light!– Scan it across the surface of the object!– This is a very precise version of structured light scanning!
Digital Michelangelo Project http://graphics.stanford.edu/projects/mich/
Source: S. Seitz
![Page 65: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/65.jpg)
The Digital Michelangelo Project, Levoy et al.!Source: S. Seitz
Laser scanning
![Page 66: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/66.jpg)
Light source! O!
J. Bouguet & P. Perona, 99!
- 1 camera,1 light source!- very cheap setup!- calibrated the light source!
Slide source: S. Savarese.!
Active Stereo: Shadows
![Page 67: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/67.jpg)
Slide source: S. Savarese.!
Active Stereo: Shadows
![Page 68: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/68.jpg)
- Dense reconstruction !- Correspondence problem again!- Get around it by using color codes!
projector! O!
L. Zhang, B. Curless, and S. M. Seitz 2002!S. Rusinkiewicz & Levoy 2002 !
Slide source: S. Savarese.!
Active Stereo: Color-Coded Stripes
![Page 69: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/69.jpg)
L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002
L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002!
Slide source: S. Savarese.!
![Page 70: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/70.jpg)
L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002
L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002!
Rapid shape acquisition: Projector + stereo cameras!
Slide source: S. Savarese.!
![Page 71: Stereo Vision (Correspondences)jjcorso/t/598F14/files/... · – For any point in one image, there should be at most one matching point in the other image! • Ordering! – Corresponding](https://reader033.fdocuments.net/reader033/viewer/2022050216/5f62024c11c2eb63b5346429/html5/thumbnails/71.jpg)
Next Lecture: Affine Structure from Motion
• Readings: FP 8.1-2; SZ 7 !
71