Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation...
-
Upload
alexandrina-hawkins -
Category
Documents
-
view
228 -
download
0
Transcript of Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation...
![Page 1: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/1.jpg)
Bahadir K. Gunturk 1
Phase Correlation
![Page 2: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/2.jpg)
Bahadir K. Gunturk 2
Phase Correlation
Take cross correlation
Take inverse Fourier transform
Location of the impulse function gives the translation amount between the images
![Page 3: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/3.jpg)
Bahadir K. Gunturk 3
Phase Correlation
![Page 4: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/4.jpg)
Computer Vision
Stereo Vision
![Page 5: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/5.jpg)
Bahadir K. Gunturk 5
Coordinate Systems
Let O be the origin of a 3D coordinate system spanned by the unit vectors i, j, and k orthogonal to each other.
i
j
kO
Px OP i
��������������
y OP j��������������
z OP k��������������
OP x y z i j k��������������
x
y
z
PCoordinate vector
![Page 6: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/6.jpg)
Bahadir K. Gunturk 6
Homogeneous Coordinatesn
a
b
c
HH
O
P
x
y
z
P
0HP OH ����������������������������
2 2 2( ) 0ax by cz a b c
0ax by cz d
0
1
x
ya b c d
z
Homogeneous coordinates 0 H P
P
TH
![Page 7: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/7.jpg)
Bahadir K. Gunturk 7
Coordinate System Changes
Translation
BPAP
![Page 8: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/8.jpg)
Bahadir K. Gunturk 8
Coordinate System Changes
Rotation
where
Exercise: Write the rotation matrix for a 2D coordinate system.
ˆ
ˆ
ˆ
AX B
AY B
AZ B
B i P
B j P
B k P
![Page 9: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/9.jpg)
Bahadir K. Gunturk 9
Coordinate System Changes
Rotation + Translation
3 3 3 1
1 3
''
''
0 1 ''
1 1
x xx x
R ty yy R y t
z zz z
![Page 10: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/10.jpg)
Bahadir K. Gunturk 10
Perspective Projection
Perspective projection equations
' ' 'x y z
x y z
![Page 11: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/11.jpg)
Bahadir K. Gunturk 11
Review: Pinhole Camera
![Page 12: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/12.jpg)
Bahadir K. Gunturk 12
Review: Perspective Projection
' ' 'x y f
x y z
![Page 13: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/13.jpg)
Bahadir K. Gunturk 13
Multi-View Geometry
Relates
• 3D World Points
• Camera Centers
• Camera Orientations
• Camera Parameters
• Image Points
![Page 14: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/14.jpg)
Bahadir K. Gunturk 14
Stereo
scene pointscene point
optical centeroptical center
image planeimage plane
p p’
![Page 15: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/15.jpg)
Bahadir K. Gunturk 15
Finding Correspondences
p p’
![Page 16: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/16.jpg)
Bahadir K. Gunturk 16
Three Questions
Correspondence geometry: Given an image point p in the first view, how does this constrain the position of the corresponding point p’ in the second?
Camera geometry (motion): Given a set of corresponding image points {pi ↔ p’i}, i=1,…,n, what are the cameras C and C’ for the two views? Or what is the geometric transformation between the views?
Scene geometry (structure): Given corresponding image points pi ↔ p’i and cameras C, C’, what is the position of the point X in space?
![Page 17: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/17.jpg)
Bahadir K. Gunturk 17
Stereo Constraints
X1
Y1
Z1
O1
Image plane
Focal plane
M
p p’
Y2
X2
Z2O2
Epipolar Line
Epipole
![Page 18: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/18.jpg)
Bahadir K. Gunturk 18
Epipolar Constraint
![Page 19: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/19.jpg)
Bahadir K. Gunturk 19
From Geometry to Algebra
O O’
P
pp’
All vectors shown lie on the same plane.
![Page 20: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/20.jpg)
Bahadir K. Gunturk 20
From Geometry to Algebra
O O’
P
pp’
( , ,1)[ ( ')] 0 with
' ( ', ',1)
T
T
p u vp t Rp
p u v
![Page 21: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/21.jpg)
Bahadir K. Gunturk 21
Matrix form of cross product
20
0
0
y z z y z
z x x z z x
x y y z y x
a b a b a a
a b a b a b a a b a b
a b a b a a
( ) 0
( ) 0
a a b
b a b
a×b=|a||b|sin(η)u a=axi+ayj+azk
b=bxi+byj+bzk
![Page 22: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/22.jpg)
Bahadir K. Gunturk 22
The Essential Matrix
( , ,1)[ ( ')] 0 with
' ( ', ',1)
T
T
p u vp t Rp
p u v
' 0Tp Ep
' 0 with Tp Ep E t R Essential matrix
![Page 23: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/23.jpg)
Bahadir K. Gunturk 23
Stereo Vision
Two cameras. Known camera positions. Recover depth.
![Page 24: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/24.jpg)
Bahadir K. Gunturk 24
Recovering Depth Information
OO22
P’P’22=Q’=Q’22
PP
OO11
P’P’11Q’Q’11
Depth can be recovered with two images and triangulation.
![Page 25: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/25.jpg)
Bahadir K. Gunturk 25
A Simple Stereo System
Zw=0
LEFT CAMERA
Left image:reference
Right image:target
RIGHT CAMERA
Elevation Zw
disparity
Depth Z
baseline
![Page 26: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/26.jpg)
Bahadir K. Gunturk 26
Stereo View
Left View Right View
Disparity
![Page 27: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/27.jpg)
Bahadir K. Gunturk 27
Stereo Disparity The separation between two matching objects
is called the stereo disparity.
![Page 28: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/28.jpg)
Bahadir K. Gunturk 28
Parallel Cameras
ZT
fZxxTlr
OOll OOrr
PP
ppll pprr
TT
ZZ
xxll xxrr
ff
T is the stereo baseline
rlxx
TfZ
rlxxd Disparity:
![Page 29: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/29.jpg)
Bahadir K. Gunturk 31
Finding Correspondences
![Page 30: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/30.jpg)
Bahadir K. Gunturk 32
Correlation Approach
For Each point (xl, yl) in the left image, define a window centered at the point
(xl, yl)LEFT IMAGE
![Page 31: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/31.jpg)
Bahadir K. Gunturk 33
Correlation Approach
… search its corresponding point within a search region in the right image
(xl, yl)RIGHT IMAGE
![Page 32: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/32.jpg)
Bahadir K. Gunturk 34
Correlation Approach
… the disparity (dx, dy) is the displacement when the correlation is maximum
(xl, yl)dx(xr, yr)RIGHT IMAGE
![Page 33: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/33.jpg)
Bahadir K. Gunturk 35
Stereo correspondence
Epipolar Constraint Reduces correspondence problem to 1D search along epipolar lines
epipolar planeepipolar lineepipolar lineepipolar lineepipolar line
![Page 34: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/34.jpg)
Bahadir K. Gunturk 36
For each epipolar lineFor each pixel in the left image
• Compare with every pixel on same epipolar line in right image
• Pick pixel with the minimum matching error
Of course, matching single pixels won’t work; so, we match regions around pixels.
Stereo correspondence
![Page 35: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/35.jpg)
Bahadir K. Gunturk 37
Comparing Windows ==??
ff gg
MostMostpopularpopular
For each window, match to closest window on epipolar line in other image.
![Page 36: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/36.jpg)
Bahadir K. Gunturk 38
Maximize Cross correlation
Minimize Sum of Squared Differences
Comparing Windows
![Page 37: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/37.jpg)
Bahadir K. Gunturk 39
Feature-based correspondence Features most commonly used:
Corners Similarity measured in terms of:
surrounding gray values (SSD, Cross-correlation) location
Edges, Lines Similarity measured in terms of:
orientation contrast coordinates of edge or line’s midpoint length of line
![Page 38: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/38.jpg)
Bahadir K. Gunturk 40
Feature-based Approach
For each feature in the left image…
LEFT IMAGE
corner line
structure
![Page 39: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/39.jpg)
Bahadir K. Gunturk 41
Feature-based Approach
Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum
RIGHT IMAGE
corner line
structure
![Page 40: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/40.jpg)
Bahadir K. Gunturk 42
Correspondence Difficulties Why is the correspondence problem difficult?
Some points in each image will have no corresponding points in the other image.(1) the cameras might have different fields of view.
(2) due to occlusion.
A stereo system must be able to determine the image parts that should not be matched.
![Page 41: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/41.jpg)
Bahadir K. Gunturk 43
Structured Light Structured lighting
Feature-based methods are not applicable when the objects have smooth surfaces (i.e., sparse disparity maps make surface reconstruction difficult).
Patterns of light are projected onto the surface of objects, creating interesting points even in regions which would be otherwise smooth.
Finding and matching such points is simplified by knowing the geometry of the projected patterns.
![Page 42: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/42.jpg)
Bahadir K. Gunturk 44
Stereo results
Ground truthScene
Data from University of Tsukuba
(Seitz)
![Page 43: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/43.jpg)
Bahadir K. Gunturk 45
Results with window correlation
Estimated depth of field(a fixed-size window)
Ground truth
(Seitz)
![Page 44: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/44.jpg)
Bahadir K. Gunturk 46
Results with better method
A state of the art methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,
International Conference on Computer Vision, September 1999.
Ground truth
(Seitz)
![Page 45: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/45.jpg)
Bahadir K. Gunturk 47
Window size
W = 3 W = 20
Better results with adaptive window• T. Kanade and M. Okutomi,
A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.
• D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998
Effect of window size
(Seitz)
![Page 46: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/46.jpg)
Bahadir K. Gunturk 48
Other constraints
It is possible to put some constraints. For example: smoothness. (Disparity usually doesn’t
change too quickly.)
![Page 47: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/47.jpg)
Bahadir K. Gunturk 49
Parameters of a Stereo System Intrinsic Parameters
Characterize the transformation from camera to pixel coordinate systems of each camera
Focal length, image center, aspect ratio
Extrinsic parameters Describe the relative
position and orientation of the two cameras
Rotation matrix R and translation vector T
pl
pr
P
Ol Or
Xl
Xr
Pl Pr
fl fr
Zl
Yl
Zr
Yr
R, T
![Page 48: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/48.jpg)
Bahadir K. Gunturk 50
Applications
courtesy of Sportvision
First-down line
![Page 49: Bahadir K. Gunturk1 Phase Correlation Bahadir K. Gunturk2 Phase Correlation Take cross correlation Take inverse Fourier transform Location of the impulse.](https://reader035.fdocuments.net/reader035/viewer/2022062803/56649f475503460f94c6927c/html5/thumbnails/49.jpg)
Bahadir K. Gunturk 51
ApplicationsVirtual advertising
courtesy of Princeton Video Image