Post on 20-Apr-2018
EECS 442 Computer vision
Stereo systems
Stereo visionRectificationCorrespondence problemActive stereo vision systems
Reading: [HZ] Chapter: 11[FP] Chapter: 11
Stereo vision
Goal: estimate the position of P given the observation ofP from two view pointsAssumptions: known camera parameters and position (K, R, T)
p
P
p
O1 O2
Stereo vision
Subgoals: - Solve the correspondence problem- Use corresponding observations to triangulate
p
P
p
O1 O2
Correspondence problem
p
P
p
O1 O2
Given a point in 3d, discover corresponding observations in left and right images [also called binocular fusion problem]
Triangulation
Intersecting the two lines of sight gives rise to P
p
P
p
O1 O2
Epipolar Plane Epipoles e1, e2
Epipolar Lines
Baseline
Epipolar geometry
O1 O2
p
P
p
e1 e2
= intersections of baseline with image planes = projections of the other camera center= vanishing points of camera motion direction
O O
P
e2p pe2
Parallel image planes
K Kx
z
v = v
y
Rectification: making two images parallel
u
v
u
v
Parallel image planes
== 00
00000
][T
TRtEK1=K2 = known
x parallel to O1O2
O O
P
e2p pe2
K Kx
z
y u
v
u
v
v = v?
Parallel image planes
( ) ( ) vTTvvTTvuv
u
TTvu ==
=
0
010
10000
0001
O O
P
e2p pe2
K Kx
z
y u
v
u
v
0pEpT =
Making image planes parallel
p
P
p
O O
H
GOAL: Estimate the perspective transformation Hthat makes the images parallel
Projective transformation
ix ix
H
ii xHx =
Now we dont have the destination image
Making image planes parallel
p
P
p
O O
H
GOAL: Estimate the perspective transformation Hthat makes the images parallel
Impose v=v How? Map e to infinity
Making image planes parallel
T21
T ]1ee[TRKe == TKe =
Making image planes parallel
p
P
p
e eK K
R,T
[ ] PTRKP [ ] P0IKP
O O
Making image planes parallel
1. Map e to the x-axis at location [1,0,1]T
[ ]T101HH1 TRH = = T21 ]1ee[e
p
P
p
e eK K
R,T
[ ] P0IKP
O O
[ ] PTRKP
Making image planes parallel
2. Send epipole to infinity
pp
[ ]T101e =
[ ]T001e =
=
101010001
H2
P
[ ]T001
e
O O
Minimizes the distortion in a neighborhood (approximates id. mapping)
Making image planes parallel
4. Align epipolar lines
pp
H = H1 H2
P
e
O O
e
3. Define: H = H1 H2
Note: H is not unique!
Projective transformation of a line (in 2D)
lHl T=
=
bvtA
H
Making image planes parallel
pp
H = H1 H2
P
e
O O
e
4. Align epipolar lines
3. Define: H = H1 H2
lHlH TT =
ll
H, H called matched pair of transformation
[HZ] Chapters: 11 (sec. 11.12)
Making image planes parallel
H
Courtesy figure S. Lazebnik
Why rectification is useful?
Makes the correspondence problem easier Makes triangulation easy
Computing depth
O O
P
e2p pe2
K Kx
z
y u
v
u
v
f
x x
Baseline B
z
O O
P
f
zfBxx =
Note: Disparity is inversely proportional to depth
Computing depth
= disparity
Disparity maps
http://vision.middlebury.edu/stereo/Disparity map / depth map
Stereo pair
Disparity map with occlusions
x x
zfBxx =
Stereo systems
Stereo visionRectificationCorrespondence problemActive stereo vision systems
Correspondence problem
p
P
p
O1 O2
Given a point in 3d, discover corresponding observations in left and right images [also called binocular fusion problem]
A Cooperative Model (Marr and Poggio, 1976)
Correlation Methods (1970--)
Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)
Correspondence problem
[FP] Chapters: 11
Correlation Methods (1970--)
Slide the window along the epipolar line until ww is maximized.
courtesy slide to J. Ponce
Correlation Methods (1970--)
Normalized Correlation; minimize:)ww)(ww()ww)(ww(
Left Right
scanline
Correlation methods
Norm. corrCredit slide S. Lazebnik
Smaller window- More detail- More noise
Larger window- Smoother disparity maps- Less prone to noise
W = 3 W = 20
Correlation methods
Credit slide S. Lazebnik
Issues
Fore shortening effect
Occlusions
OO
O O
It is desirable to have small B/z ratio!
O O
Issues
O O
small B/z ratio
Issues
Small error in measurements implies large error in estimating depth
Homogeneous regions
Issues
Hard to match pixels in these regions
Repetitive patterns
Issues
Correspondence problem is difficult!
- Occlusions- Fore shortening- Baseline trade-off- Homogeneous regions- Repetitive patterns
Apply non-local constraints to help enforce the correspondences
Results with window search
Data
Ground truth
Credit slide S. Lazebnik
Window-based matching
Improving correspondence: Non-local constraints
Uniqueness For any point in one image, there should be at most one
matching point in the other image
courtesy slide to J. Ponce
Uniqueness For any point in one image, there should be at most one
matching point in the other image
Ordering Corresponding points should be in the same order in both
views
Improving correspondence: Non-local constraints
courtesy slide to J. Ponce
Uniqueness For any point in one image, there should be at most one
matching point in the other image
Ordering Corresponding points should be in the same order in both
views
Not alwaysin presenceof occlusions!
courtesy slide to J. Ponce
Improving correspondence: Non-local constraints
Dynamic Programming (Baker and Binford, 1981)
Nodes = matched feature points (e.g., edge points). Arcs = matched intervals along the epipolar lines. Arc cost = discrepancy between intervals.
Find the minimum-cost path going monotonicallydown and right from the top-left corner of thegraph to its bottom-right corner.
[Uses ordering constraint]
courtesy slide to J. Ponce
Dynamic Programming (Baker and Binford, 1981)
courtesy slide to J. Ponce
Dynamic Programming (Ohta and Kanade, 1985)
Reprinted from Stereo by Intra- and Intet-Scanline Search, by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and MachineIntelligence, 7(2):139-154 (1985). 1985 IEEE.
Uniqueness For any point in one image, there should be at most one
matching point in the other image
Ordering Corresponding points should be in the same order in both
views
Smoothness Disparity is typically a smooth function of x (expect in occluding
boundaries)
Improving correspondence: Non-local constraints
Smoothness
Stereo matching as energy minimization
I1
W1(i)
I2
W2(i+D(i))
D
D(i)
Energy functions of this form can be minimized using graph cuts
Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 01
)(),,( smooth21data DEDIIEE +=
( ) =ji
jDiDE,neighbors
smooth )()(( )221data ))(()( +=i
iDiWiWE
Credit slide S. Lazebnik
Graph cutsGround truth Window-based matching
Stereo matching as energy minimization
Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 01
http://www.middlebury.edu/stereo/
Two-frame stereo correspondence algorithms
Click here
http://www.middlebury.edu/stereo/
stereo vision software development kit.Stereo SDKA. Criminisi, A. Blake and D. Robertson
Application foreground/background separationV. Kolmogorov, A. Criminisi, A. Blake, G. Cross and C. Rother. Bi-layer segmentation of binocular stereo video CVPR 2005
http://research.microsoft.com/~antcrim/demos/ACriminisi_Recognition_CowDemo.wmv
Click on ACriminisi_i2i_BilayerSegmentation.wmv
http://research.microsoft.com/users/antcrim/papers/Criminisi_cvpr2005.pdf
Application 3D Urban Scene Modeling3D Urban Scene Modeling Integrating Recognition and Reconstruction,N. Cornelis, B. Leibe, K. Cornelis, L. Van Gool, IJCV 08.
http://www.vision.ee.ethz.ch/showroom/index.en.html#
Click on cornelis-cognitiveloops-vpcvpr06.avi
Stereo systems
Stereo visionRectificationCorrespondence problemActive stereo vision systems
Active stereo (point)
Replace one of the two cameras by a projector- Single camera - Projector geometry calibrated- Correspondence problem solved!
P
projector O2
p
Active stereo (stripe)
projectorO
-Projector and camera are parallel- Correspondence problem solved!
Laser scanning
Optical triangulation Project a single stripe of laser light Scan it across the surface of the object This is a very precise version of structured light scanning
Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/
Source: S. Seitz
The Digital Michelangelo Project, Levoy et al.Source: S. Seitz
Laser scanning
Light source O
Active stereo (shadows)J. Bouguet & P. Perona, 99
- 1 camera,1 light source- very cheap setup- calibrated the light source
Active stereo (shadows)
Click here
Active stereo (dense)
- Dense reconstruction - Correspondence problem again- Get around it by using color codes
projectorO
L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002
L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002
Rapid shape acquisition: Projector + stereo cameras
Next lecture
Volumetric stereo
Human Stereopsis
Figure from US Navy Manual of Basic Optics and Optical Instruments, prepared by Bureau of Naval Personnel. Reprinted by Dover Publications, Inc., 1969.
Credit slide J. Ponce
Human Stereopsis: Reconstruction
Disparity: d = r-l = D-F;
d=0
d
What if F is not known?
Helmoltz (1909):
There is evidence showing the vergence anglescannot be measured precisely.
Humans get fooled by bas-relief sculptures.
Human Stereopsis: Reconstruction
Credit slide J. Ponce
Human Stereopsis: Binocular Fusion
Credit slide J. Ponce
How are the correspondences established?
Julesz (1971): Is the mechanism for binocular fusiona monocular process or a binocular one?? There is anecdotal evidence for the latter (camouflage).
Random dot stereograms provide an objective answer
Human Stereopsis: Binocular Fusion
Credit slide J. Ponce
EECS 442 Computer visionStereo systems Stereo systems Correlation methodsCorrelation methodsResults with window searchImproving correspondence: Non-local constraintsImproving correspondence: Non-local constraintsImproving correspondence: Non-local constraintsImproving correspondence: Non-local constraintsSmoothnessStereo matching as energy minimizationStereo matching as energy minimizationStereo systems Laser scanningLaser scanningNext lecture