Marc-Antoine Drouin, Martin Trudeau and Sebastien Roy´drouim/pub/drouin-stereocvpr05-poster.… ·...
Transcript of Marc-Antoine Drouin, Martin Trudeau and Sebastien Roy´drouim/pub/drouin-stereocvpr05-poster.… ·...
Geo-consistency for Wide Multi-Camera StereoMarc-Antoine Drouin, Martin Trudeau and Sebastien Roy
Departement d’Informatique et recherche operationnelle, Universite de Montreal, Canadaemail: {drouim,trudeaum,roys}@iro.umontreal.ca
Problem: 3D Reconstruction of a scene
from a reference camera using multiple
supporting ones
For each pixel p we need to choose
•a depth f(p)
•a mask of used cameras g(p)
so as to minimize the energy
E(f, g) =∑
p∈Pe(p, f(p), g(p))
+smoothing.
Handling Visibility
Exact wheng(p) = V (p|f(p), f)
(the real visibility)
•Long range interaction ⇒ en-ergy minimization is hard
Heuristic wheng(p) = arg minm∈M e(p, f(p), m)
(M is a set of plausible masks)
•Photo-consistency �⇒ correctvisibility
Geo-consistency
Defined as
g(p) ≤ V (p|f(p), f)
•Occluded cameras are not used
•Visible cameras are not alwaysused
Justified by Nakamura96:
•Using an occluded camera ⇒important artifacts
•Not using a visible camera ⇒no significant artifacts
Bias in Border Localization
Closest objects are enlarged by stan-dard algorithms.
Supportingcamera
Referencecamera
GT occludeesGT occluders
DM occludeesDM occluders
1 2 3 4 5
•Large number of occluders indepth map are occludees accord-ing to ground truth (zone 3)
•Visibility from initial depth mapis useless to iteratively improve thesolution
Pseudo-Visibility
It compensates for the bias by label-ing both occluders and occludees asinvisible.
•Preserves Geo-consistency
•Computed using rendering tech-niques (the depth map is repre-sented as a continuous mesh)
Algorithm: Overview
• Initialize with all cameras visiblein each mask
•Compute depth map
•Update pseudo-visibility masksusing depth map
• Iterate until convergence
depth map f
initial masks g
stereo matcher
Pseudo-Visibility
final depth map f
final masks g
masks changed ?
new masks g
History
Yes
No
stereo matcher
Ordering Constraint
It is respected when the order inwhich 2 objects are encounteredalong an epipolar line does notchange.
•Not always true
•Continuous mesh ⇒ordering constraint is respected
2
1
2 1 2 1
2
1
12 21
History
Once a camera is removed, it isnever used again.
•Guaranteed convergence to aGeo-consistent solution that re-spects the ordering constraint
Results:Middlebury
An error is a difference greater than1 label from the ground truth.
Middlebury sequencealgorithmsbarn1 barn2 bull postervenussawtoothBoykov99 3.5 % 3.1 % 0.7 % 3.7 % 3.4 % 3.3%Ours 0.8 %0.6 %0.4 %1.1 %2.4 % 1.1 %Kang01 1.4 % 1.5 % 0.9 %1.1 %4.0 % 1.5%Drouin05 0.7 %3.9 % 0.8 % 4.0 % 5.3% 1.0 %
Results:Baseline Test
Depth maps recovered for the sameviewpoint with different baselinesshould be identical.
•baseline ↑ ⇒ occlusion ↑
0 2 4 6 8 10 12 14%
Boykov99Sanfourche04Kolmogorov02Sanfourche04
Ours 1x vs 2x2x vs 3x3x vs 4x1x vs 4x
% differing pixels
Ours (3x)Reference
Kang01 (3x) Kolmogorov02 (3x)
Conclusion
• New framework to model occlu-sion in stereo by introducing Geo-consistency
•Occlusion modeling is added tostandard stereo algorithms
Future Work
•Better handling of regions break-ing the ordering constraint
•Extending the framework to vol-umetric reconstruction