1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir...
-
Upload
jessica-haynes -
Category
Documents
-
view
218 -
download
0
Transcript of 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir...
![Page 1: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/1.jpg)
1
Interest Operators
• Harris Corner Detector: the first and most basic interest operator
• Kadir Entropy Detector and its use in object recognition
• SIFT interest point detector and region descriptor
• MSER region detector and Harris Affine in region matching
All lectures are from posted research papers.
![Page 2: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/2.jpg)
2
0. Introduction to Interest Operators
• Find “interesting” pieces of the image– e.g. corners, salient regions – Focus attention of algorithms– Speed up computation
• Many possible uses in matching/recognition– Search– Object recognition– Image alignment & stitching– Stereo– Tracking– …
![Page 3: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/3.jpg)
3
Interest points
0D structurenot useful for matching
1D structureedge, can be localised in 1D, subject to the aperture problem
2D structurecorner, or interest point, can be localised in 2D, good for matching
Interest Points have 2D structure.
![Page 4: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/4.jpg)
4
Local invariant photometric descriptors
( )local descriptor
Local : robust to occlusion/clutter + no segmentation
Photometric : (use pixel values) distinctive descriptions
Invariant : to image transformations + illumination changes
![Page 5: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/5.jpg)
5
History - Matching
1. Matching based on correlation alone 2. Matching based on geometric primitives
e.g. line segments
Not very discriminating (why?)
Solution : matching with interest points & correlation [ A robust technique for matching two uncalibrated images through
the recovery of the unknown epipolar geometry, Z. Zhang, R. Deriche, O. Faugeras and Q. Luong, Artificial Intelligence 1995 ]
![Page 6: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/6.jpg)
6
Zhang Approach
• Extraction of interest points with the Harris detector
• Comparison of points with cross-correlation
• Verification with the fundamental matrix
The fundamental matrix maps points from the first image to
corresponding points in the second matrix using a homography
that is determined through the solution of a set of equations
that usually minimizes a least square error. CH 12-13
![Page 7: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/7.jpg)
7
Preview: Harris detector
Interest points extracted with Harris (~ 500 points)
![Page 8: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/8.jpg)
8
Harris detector
![Page 9: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/9.jpg)
9
Harris detector
![Page 10: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/10.jpg)
10
Cross-correlation matching
Initial matches – motion vectors (188 pairs)
Match two
points based
on how similar
their neighbor-
hoods are.
![Page 11: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/11.jpg)
11
Global constraints
Robust estimation of the fundamental matrix (RANSAC)
99 inliers 89 outliers
![Page 12: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/12.jpg)
12
General Interest Detector/Descriptor Approach
1) Extraction of interest points (characteristic locations)
2) Computation of local descriptors (rotational invariants)
3) Determining correspondences
4) Selection of similar images
( )local descriptor
interest points
![Page 13: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/13.jpg)
13
1. Harris detector
Based on the idea of auto-correlation
Important difference in all directions => interest point
![Page 14: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/14.jpg)
14
Background: Moravec Corner Detector
w
• take a window w in the image• shift it in four directions
(1,0), (0,1), (1,1), (-1,1)• compute a difference for each• compute the min difference at
each pixel• local maxima in the min image
are the corners
E(x,y) = ∑ w(u,v) |I(x+u,y+v) – I(u,v)|2 u,v in w
![Page 15: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/15.jpg)
15
Shortcomings of Moravec Operator
• Only tries 4 shifts. We’d like to consider “all” shifts.
• Uses a discrete rectangular window. We’d like to use a smooth circular (or later elliptical) window.
• Uses a simple min function. We’d like to characterize variation with respect to direction.
Result: Harris Operator
![Page 16: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/16.jpg)
16
Harris detector
2
),(
)),(),((),( yyxxIyxIyxf kkkWyx
k
kk
y
xyxIyxIyxIyyxxI kkykkxkkkk )),(),((),(),(with
2
),(
),(),(),(
Wyx
kkykkx
kky
xyxIyxIyxf
Auto-correlation fn (SSD) for a point and a shift ),( yx ),( yx
Discrete shifts can be avoided with the auto-correlation matrix
what is this?
SSD means summed square difference
![Page 17: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/17.jpg)
17
Harris detector
Rewrite as inner (dot) product
The center portion is a 2x2 matrixHave we seen
this matrix before?
![Page 18: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/18.jpg)
18
Harris detector
y
x
yxIyxIyxI
yxIyxIyxI
yx
Wyxkky
Wyxkkykkx
Wyxkkykkx
Wyxkkx
kkkk
kkkk
),(
2
),(
),(),(
2
)),((),(),(
),(),()),((
Auto-correlation matrix M
W (x,y)
![Page 19: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/19.jpg)
19
Harris detection
• Auto-correlation matrix– captures the structure of the local neighborhood– measure based on eigenvalues of M
• 2 strong eigenvalues => interest point
• 1 strong eigenvalue => contour
• 0 eigenvalue => uniform region
• Interest point detection– threshold on the eigenvalues– local maximum for localization
![Page 20: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/20.jpg)
20
Harris Corner Detector
• Corner strength R = Det(M) – k Tr(M)2
• Let and be the two eigenvalues• Tr(M) = + • Det(M) =
• R is positive for corners, negative for edges, and small for flat regions
• Selects corner pixels that are 8-way local maxima
R = Det(M) – k Tr(M)2 is the Harris Corner Detector.
![Page 21: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/21.jpg)
21
Now we need a descriptor
Vector comparison using a distance measure
( ) ( )=?
What are some suitable distance measures?
![Page 22: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/22.jpg)
22
Distance Measures
• We can use the sum-square difference of the values of the pixels in a square neighborhood about the points being compared.
SSD = ∑∑(W1(i,j) –(W2(i,j))2
W1 W2
Works when the motion is mainly a translation.
![Page 23: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/23.jpg)
23
Some Matching Results from Matt Brown
![Page 24: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/24.jpg)
24
Some Matching Results
![Page 25: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/25.jpg)
25
Summary of the approach
• Basic feature matching = Harris Corners & Correlation• Very good results in the presence of occlusion and clutter
– local information– discriminant greyvalue information – invariance to image rotation and illumination
• Not invariance to scale and affine changes
• Solution for more general view point changes– local invariant descriptors to scale and rotation– extraction of invariant points and regions
![Page 26: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/26.jpg)
26
Rotation/Scale Invariance
Translation Rotation Scale
Is Harris invariant?
? ? ?
Is correlation invariant?
? ? ?
original translated rotated scaled
![Page 27: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/27.jpg)
27
Rotation/Scale Invariance
Translation Rotation Scale
Is Harris invariant?
? ? ?
Is correlation invariant?
? ? ?
original translated rotated scaled
![Page 28: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/28.jpg)
28
Rotation/Scale Invariance
Translation Rotation Scale
Is Harris invariant?
YES ? ?
Is correlation invariant?
? ? ?
original translated rotated scaled
![Page 29: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/29.jpg)
29
Rotation/Scale Invariance
Translation Rotation Scale
Is Harris invariant?
YES YES ?
Is correlation invariant?
? ? ?
original translated rotated scaled
![Page 30: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/30.jpg)
30
Rotation/Scale Invariance
Translation Rotation Scale
Is Harris invariant?
YES YES NO
Is correlation invariant?
? ? ?
original translated rotated scaled
![Page 31: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/31.jpg)
31
Rotation/Scale Invariance
Translation Rotation Scale
Is Harris invariant?
YES YES NO
Is correlation invariant?
? ? ?
original translated rotated scaled
![Page 32: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/32.jpg)
32
Rotation/Scale Invariance
Translation Rotation Scale
Is Harris invariant?
YES YES NO
Is correlation invariant?
YES ? ?
original translated rotated scaled
![Page 33: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/33.jpg)
33
Rotation/Scale Invariance
Translation Rotation Scale
Is Harris invariant?
YES YES NO
Is correlation invariant?
YES NO ?
original translated rotated scaled
![Page 34: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/34.jpg)
34
Rotation/Scale Invariance
Translation Rotation Scale
Is Harris invariant?
YES YES NO
Is correlation invariant?
YES NO NO
original translated rotated scaled
![Page 35: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/35.jpg)
35
Matt Brown’s Invariant Features
• Local image descriptors that are invariant (unchanged) under image transformations
![Page 36: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/36.jpg)
36
Canonical Frames
![Page 37: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/37.jpg)
37
Canonical Frames
![Page 38: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/38.jpg)
38
Multi-Scale Oriented Patches
• Extract oriented patches at multiple scales using dominant orientation
![Page 39: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/39.jpg)
39
Multi-Scale Oriented Patches
• Sample scaled, oriented patch
![Page 40: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/40.jpg)
40
Multi-Scale Oriented Patches
• Sample scaled, oriented patch– 8x8 patch, sampled at 5 x scale
![Page 41: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/41.jpg)
41
• Sample scaled, oriented patch– 8x8 patch, sampled at 5 x scale
• Bias/gain normalized (subtract the mean of a patch and divide by the variance to normalize)– I’ = (I – )/
Multi-Scale Oriented Patches
8 pixels
40 pixels
![Page 42: 1 Interest Operators Harris Corner Detector: the first and most basic interest operator Kadir Entropy Detector and its use in object recognition SIFT interest.](https://reader030.fdocuments.net/reader030/viewer/2022032313/56649e695503460f94b65da0/html5/thumbnails/42.jpg)
42
Matching Interest Points: Summary
• Harris corners / correlation– Extract and match repeatable image features– Robust to clutter and occlusion– BUT not invariant to scale and rotation
• Multi-Scale Oriented Patches– Corners detected at multiple scales– Descriptors oriented using local gradient
• Also, sample a blurred image patch
– Invariant to scale and rotation
Leads to: SIFT – state of the art image features