CS 223b1

More on stereo and More on stereo and correspondencecorrespondence

CS 223b2

==??

ff gg

MostMostpopularpopular

For each window, match to closest window on epipolar line in other image.

(slides O. Camps)

Comparing Windows:Comparing Windows:

CS 223b3

Window sizeWindow size

W = 3 W = 20

Better results with adaptive window• T. Kanade and M. Okutomi,

A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.

• D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998

Effect of window size

(S. Seitz)

CS 223b4

Dynamic Programming (Baker and Dynamic Programming (Baker and Binford, 1981)Binford, 1981)

Find the minimum-cost path going monotonicallydown and right from the top-left corner of thegraph to its bottom-right corner.

• Nodes = matched feature points (e.g., edge points).• Arcs = matched intervals along the epipolar lines.• Arc cost = discrepancy between intervals.

CS 223b5

Dynamic Programming (Baker and Dynamic Programming (Baker and Binford, 1981)Binford, 1981)

Find the minimum-cost path going monotonicallydown and right from the top-left corner of thegraph to its bottom-right corner.

• Nodes = matched feature points (e.g., edge points).• Arcs = matched intervals along the epipolar lines.• Arc cost = discrepancy between intervals.

CS 223b6

Dynamic Programming (Ohta and Kanade, Dynamic Programming (Ohta and Kanade, 1985)1985)

Reprinted from “Stereo by Intra- and Intet-Scanline Search,” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and MachineIntelligence, 7(2):139-154 (1985). 1985 IEEE.

CS 223b7

Other constraintsOther constraints

Smoothness: disparity usually doesn’t change too quickly.

Unfortunately, this makes the problem 2D again.

Solved with a host of graph algorithms, Markov Random Fields, Belief Propagation, ….

Uniqueness constraint (each feature can at most have one match)

Occlusion and disparity are connected.

CS 223b8

Feature-based MethodsFeature-based Methods

Conceptually very similar to Correlation-based methods, but: They only search for correspondences of a sparse set of image features.

Correspondences are given by the most similar feature pairs.

Similarity measure must be adapted to the type of feature used.

CS 223b9

Feature-based Methods:Feature-based Methods:

Features most commonly used: Corners

Similarity measured in terms of: surrounding gray values (SSD, Cross-correlation) location

Edges, Lines Similarity measured in terms of:

orientation contrast coordinates of edge or line’s midpoint length of line

CS 223b10

Example: Comparing linesExample: Comparing lines

ll and lr: line lengths l and r: line orientations (xl,yl) and (xr,yr): midpoints cl and cr: average contrast along lines l m c : weights controlling influence

The more similar the lines, the larger S is!

CS 223b11

Correspondence By FeaturesCorrespondence By Features

RIGHT IMAGE

corner line

structure

Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum

CS 223b12

Dense Stereo Matching: ExamplesDense Stereo Matching: Examples

View extrapolation results

input depth image novel view

[Matthies,Szeliski,Kanade’88]

CS 223b13

Dense Stereo MatchingDense Stereo Matching

Some other view extrapolation results

input depth imagenovel view

CS 223b14

Dense Stereo MatchingDense Stereo Matching

Compute certainty map from correlations

input depth map certainty map

CS 223b15

Ordering constraintOrdering constraint

Usually, order of points in two images is same.

Is this always true? If we match pixel i in image 1 to pixel j in image 2, no matches that follow will affect which are the best preceding matches.

Example with pixels (a la Cox et al.).

CS 223b16

The Ordering ConstraintThe Ordering Constraint

But it is not always the case ...This enables dynamic programming

Points on the epipolar lines appear in the same order

CS 223b17

Correspondence Problem 2Correspondence Problem 2

Correspondence fail for smooth surfaces

There is currently no good solution to the correspondence problem

CS 223b18

Correspondence Problem 3Correspondence Problem 3

Regions without texture Highly Specular surfaces Translucent objects

CS 223b19

Stereo Vision: OutlineStereo Vision: Outline

Basic Equations Epipolar Geometry Image Rectification Reconstruction Correspondence Active Range Imaging Technology Dense and Layered Stereo Smoothing With Markov Random Fields

CS 223b20

How can We Improve Stereo?How can We Improve Stereo?

Space-time stereo scanneruses unstructured light to aidin correspondence

Result: Dense 3D mesh (noisy)

CS 223b21

Prof Marc Levoy @ StanfordProf Marc Levoy @ Stanford

By James Davis, Honda Research,

Now UCSC

CS 223b22

rect

ified

Active Stereo (Structured Active Stereo (Structured Light)Light)

CS 223b23

DP for CorrespondenceDP for Correspondence

Does this always work? When would it fail?

Failure Example 1 Failure Example 2 Failure Example 3

CS 223b24

Stereo CorrespondencesStereo Correspondences

… …Left scanline Right scanline

Match

Match

MatchOcclusion Disocclusion

CS 223b25

Stereo CorrespondencesStereo Correspondences

… …Left scanline Right scanline

CS 223b26

Search Over CorrespondencesSearch Over Correspondences

Three cases: Sequential – cost of match Occluded – cost of no match Disoccluded – cost of no match

Left scanline

Right scanline

Occluded Pixels

Disoccluded Pixels

CS 223b27

Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path.

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal

Stereo Matching with Dynamic ProgrammingStereo Matching with Dynamic Programming

CS 223b28

Stereo Matching with Dynamic ProgrammingStereo Matching with Dynamic Programming

Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Start

End

CS 223b29

Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path.

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal

Stereo Matching with Dynamic ProgrammingStereo Matching with Dynamic Programming

CS 223b30

Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path.

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal

Stereo Matching with Dynamic ProgrammingStereo Matching with Dynamic Programming