Lecture8 - Stereo

8/2/2019 Lecture8 - Stereo

http://slidepdf.com/reader/full/lecture8-stereo 1/67

6.801/866

Stereopsis

T. Darrell



text



Outline

• Reconstruction

• Rectification

• Early vs. Late

• Window Matching• Edge Matching

• Ordering Constraint

• More views



baseline

depth



baseline

depth

Triangulate on two images of the

same point to recover depth.

– Feature matching across views

– Calibrated cameras

Left Right

Last lecture: only needto match features across

epipolar lines…



Geometric Reconstruction

Geometric: midpoint of intersection of Op and O’p’

Multiple views?



Algebraic Reconstruction

Point must satisfy both projection equations:

and therefore:

solve using least squares.

No obvious geometric interpretation; well-defined for multiple views.



Nonlinear Reconstruction

Nonlinear technique:

search for point Q with minimal projection error:

(initialize with one of previous linear methods.)



Rectified image pair

• Choose normalized image

locations

• Common image plane

• Horizontal epipolar lines

• Correspondence search on

scanlines!



Pinhole Camera Model

Focal length f

Virtual

Image

Center of

projection

Image

plane



Pinhole Camera Model

),,( Z Y X P =

Virtual

Image

f −

O y

x

z

Image

plane

),( y x p =

1

1

1 Z

X

f x−=



Basic Stereo Derivations

1O y

x

z

f −

2O y

x

z

1 p

B

),,(1 Z Y X P =

2 p

B f x x Z ,,,of functionaasfor expressionDerive 21




),,(1 Z Y X P =1O y

x

z

f −

2O y

x

z

211

1

1

1

12

1

11 ,

x x

B f Z

Z B f x

Z B X f x

Z X f x

−

=⇒

−=+−=−=

B




y

x

z

Ray from

camera 2

1 x

Ray from

camera 1

2 x

d

B f Z

x xd

=

−=21 :disparitytheDefine



Stereo Vision

baseline

depth

),(),(

y xd

B f y x Z =

Z ( x, y) is depth at pixel ( x, y)

d ( x, y) is disparity

Left Right

Matching across scan lines

How do matching errors affect

the depth error?



Outline

Reconstruction

Rectification

• Early vs. Late



• More views



Consider human vision

Differences?

• active foveation

• image plane

• object recognition



Human vision is very accurate at relative depth

judgements.

Julez (1960): which happens first, recognition or

fusion?



Random dot stereograms



Correspondence is ambiguous



Marr-Poggio (1976)

Three constraints:

• compatibility• uniqueness

• continuity

Works well on RDS….but not so well on natural images…



Outline

Reconstruction

Rectification

Early vs. Late



• More views



Correspondence using window matching

Points are highly individually ambiguous…

More unique matches are possible with small regionsof image.



Correspondence using window matching

Left Right

error

disparity

scanline

Criterion function:



Sum of Squared (Pixel) Differences

Left Right

Lw Rw

Lw

L I

Rw

R I

mm

),( L L y x ),( L L yd x −

∑∈−−=

+≤≤−+≤≤−=

),(),(

2

2222

)],(),([),,(

:disparityof functionaasdifferenceintensitythemeasurescostSSDThe

},|,{),(

:functionwindowthedefineWe

pixels.of windows byingcorrespondareand

y xW vu R Lr

mmmmm

R L

m

vd u I vu I d y xC

yv y xu xvu y xW

mmww



Image Normalization

• Even when the cameras are identical models, there can bedifferences in gain and sensitivity.

• The cameras do not see exactly the same surfaces, so their

overall light levels can differ.• For these reasons and more, it is a good idea to normalize

the pixels in each window:

pixel Normalized ),(

),(ˆ

magnitudeWindow )],([

pixelAverage ),(

),(

),(),(

2

),(

),(),(),(

1

y xW

y xW vu y xW

y xW vu y xW

m

m

m

m

m

I I

I y x I y x I

vu I I

vu I I

−

−=

=

=

∑

∑

∈

∈



Images as Vectors

“Unwrap”

image to form

vector, usingraster scan order

Left Right

Lw Rw

Lw

row 1 m

m

Lwrow 3

m row 2 m

Each window is a vector

in an m2 dimensional

vector space.

Normalization makes

them unit length.

m



Image windows as vectors



Possible metrics

Lw

)(d w R Distance?

Angle?



Correspondence Using Correlation

Left Disparity Map

Images courtesy of Point Grey Research



Foreshortening

Window methods assume fronto-parallel surface at 3-D point.

Initial estimates of the disparity can be used to warp thecorrelation windows to compensate for unequal amounts of foreshortening in the two pictures [Kass, 1987; Devernay

and Faugeras, 1994].



Outline

Reconstruction

RectificationEarly vs. Late

Window Matching

• Edge Matching


• More views

bl i h i d h d



Problems with window methods

Patch too small?

Patch too large?

Can try variable patch size [Okutomi and Kanade],or arbitrary window shapes [Veksler and Zabih]

Should match between physically meaningfulquanties, and at multiple scales [Marr]…

M P i l i h



Marr-Poggio algorithm

Search for edges, a.k.a. “zero crossings”: (more

during edge detection lectures…)

O d i i



Ordering constraint

“It is reasonable to assume that the order of matching image

features along a pair of epipolar lines is the inverse of the

order of the corresponding surface attributes along the

curve where the epipolar plane intersects the observed

object’s boundary.”

This is the so-called ordering constraint introduced by [Baker

and Binford, 1981; Ohta and Kanade, 1985].

O d i t i t



Ordering constraint

O d i t i t



Ordering constraint

Oops!

DP b d h



DP-based search

Ordering constraint == smooth path through match graph.

Consider path’s cost over a graph whose nodes correspond to pairs of left and right image features, and arcs represent

matches between left and right intensity profile intervals

bounded by the features of the corresponding nodes

S h O C d



Search Over Correspondences

Occluded Pixels

Left scanline

Right scanline

Dis-occluded Pixels

Three cases: – Sequential – cost of match

– Occluded – cost of no match

– Disoccluded – cost of no match

Dynamic Programming



Dynamic Programming

Occluded Pixels

Left scanline

Di s

- o c c l u d e d P i x e l s

Ri gh t s c a nl i n

e

End

Start

Dynamic programming

yields the optimal

path through grid.This is the best set of

matches that satisfy

the orderingconstraint

Dynamic Programming



Dynamic Programming

1=i 1 1 112Π

22Π2= j2 2 22=i

32Π

3 3 33=i

1−t C t C 1+t C

Principle of Optimality for an n-stage assignment problem:

)(maxarg)( 1 iC jC t ijit −Π=

Dynamic Programming



Dynamic Programming

1=i 1 1 1

2)2( =t b2= j2 2 22=i

3 3 33=i

1−t C t C 1+t C



Dynamic Programming



Dynamic Programming

1=i 1 1 1

2

3

2

13Π

23Π

33Π

22=i

3= j3 33=i

1−t C t C 1+t C



Dynamic Programming



Dynamic Programming

1=i 1 1 1

2

3

2

1)3( =t b

22=i

3= j3 33=i

1−t C t C 1+t C



Dynamic Programming



Dynamic Programming

1)1( =t b1= j1=i 1 1 1

2

3

2

1)3(=

t b

2)2( =t b22=i

3 33=i

1−t C t C 1+t C



Dynamic Programming



Dynamic Programming

1=i 1 1 1

2

3

2

3

2

3

2−T C 1−T C T C

Back-chaining recovers the optimal path and its cost:

...),(),(),(maxarg **

1

***

T T T T T T T jT sb s sC C jC s ===−

2=i

3=i

)2(

2*

*

T T

T

C C

s

=

=



Stereo Matching with Dynamic



Programming

Occluded Pixels

Left scanline

Di s - o c c l u d e d P i x

e l s

Ri gh t s c a nl i n e

Scan across grid

computing optimal

cost for each nodegiven its upper-left

neighbors.

Backtrack from theterminal to get the

optimal path.Terminal


i



Programming

Occluded Pixels

Left scanline


e l s


Scan across grid

computing optimal


neighbors.




P i



Programming

Occluded Pixels

Left scanline


e l s


Scan across grid

computing optimal


neighbors.



DP vs Edges



g

Edges:

DP:

• Which method is better? – Edges are more “meaningful” [Marr]…but hard to find!

– Edges tend to fail in dense texture (outdoors)

– Correlation tends to fail in smooth featureless areas

Computing Correspondences



p g p

Both methods fail for smooth surfaces

There is currently no good solution to thecorrespondence problem

Outline



Reconstruction


Window Matching

Edge Matching

Ordering Constraint

• More views

Three (calibrated) views



Trinocular Stereo ResultsWhy this camera

arrangement?



Trinocular stereo system available from Point

Gray Research for $5K (circa ’97)

More views reduce ambiguity



Middlebury stereo page



http://www.middlebury.edu/stereo/

Outline



Reconstruction


Window Matching

Edge Matching

Ordering Constraint

More views

[ Most figures adapted from Forsythe and Ponce ]

Lecture8 - Stereo

Documents

Transcript of Lecture8 - Stereo