Lecture 8: Interest Point Detection - University of...
Transcript of Lecture 8: Interest Point Detection - University of...
![Page 2: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/2.jpg)
#2
Last Lecture : Edge Detection• Preprocessing of image is desired to eliminate or
at least minimize noise effects• There is always tradeoff between sensitivity and
accuracy in edge detection • The parameters that we can set for edge detection
and labeling include the size of the edge detection mask and the value of the threshold
• A larger mask or a higher gray level threshold will tend to reduce noise effects, but may result in a loss of valid edge points
![Page 3: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/3.jpg)
Example
original image (Lena)
![Page 4: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/4.jpg)
Compute Gradients
X-Derivative of Gaussian Y-Derivative of Gaussian Gradient Magnitude
![Page 5: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/5.jpg)
Get Orientation at Each Pixel
• Threshold at minimum level• Get orientation
theta = atan2(gy, gx)
![Page 6: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/6.jpg)
#6
![Page 7: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/7.jpg)
Edge Thinning: Non-maximum suppression for each orientation
At q, we have a maximum if the value is larger than those at both p and at r. Interpolate to get these values.
![Page 8: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/8.jpg)
Today’s Lecture
Goal: Find features between multiple images taken from different position or time to be robustly matched
Approaches:• Patch/Template Matching• Interest Point Detection
![Page 9: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/9.jpg)
Applications
Comparison Between 2 or more images• Image alignment • Image Stitching• 3D reconstruction• Object recognition ( matching ) • Indexing and database retrieval• Object tracking• Robot navigation
![Page 10: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/10.jpg)
• What points would you choose?Known as: Template Matching Method, with option to use correlation as a metric
![Page 11: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/11.jpg)
Correlation#11
![Page 12: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/12.jpg)
Correlation#12
• Use Cross Correlation to find the template in an image
• Maximum indicate high similarity
![Page 13: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/13.jpg)
Template/Patch Matching Options
• What do we do for different scales of the patch ?– Compute multiple Templates at different sizes– Match for different scales of the template
• What can we do to identify the shape of the pattern at different mean brightness levels – Remove the mean of the template
• Example in 1D
![Page 14: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/14.jpg)
![Page 15: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/15.jpg)
We need more robust feature descriptors for matching !!!!!
![Page 16: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/16.jpg)
Interest Points
• Local features associated with a significant change of an image property of several properties simultaneously (e.g., intensity, color, texture).
![Page 17: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/17.jpg)
Why Extract Interest Points?
• Corresponding points (or features) between images enable the estimation of parameters describing geometric transforms between the images.
![Page 18: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/18.jpg)
What if we don’t know the correspondences?
• Need to compare feature descriptors of local patches surrounding interest points
( ) ( )=?feature
descriptorfeature
descriptor
?
![Page 19: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/19.jpg)
Properties of good features• Local: features are local, robust to occlusion and
clutter (no prior segmentation!).• Accurate: precise localization.
• Invariant / Covariant• Robust: noise, blur, compression, etc.
do not have a big impact on the feature.
• Distinctive: individual features can be matched to a large database of objects.
• Efficient: close to real-time performance.
Repeatable
![Page 20: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/20.jpg)
Invariant / Covariant
• A function f( ) is invariant under some transformation T( ) if its value does change when the transformation is applied to its argument:
if f(x) = y then f(T(x))=y
• A function f( ) is covariant when it commutes with the transformation T( ):
if f(x) = y then f(T(x))=T(f(x))=T(y)
![Page 21: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/21.jpg)
Invariance• Features should be detected despite geometric or
photometric changes in the image.• Given two transformed versions of the same
image, features should be detected in corresponding locations.
![Page 22: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/22.jpg)
Example: Panorama Stitching
• How do we combine these two images?
![Page 23: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/23.jpg)
Panorama stitching (cont’d)
Step 1: extract featuresStep 2: match features
![Page 24: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/24.jpg)
Panorama stitching (cont’d)
Step 1: extract featuresStep 2: match featuresStep 3: align images
![Page 25: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/25.jpg)
What features should we use?
Use features with gradients in at least two (significantly) different orientations patches ? Corners ?
![Page 26: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/26.jpg)
What features should we use? (cont’d)
(auto-correlation)
![Page 27: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/27.jpg)
Corners vs Edges#27
![Page 28: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/28.jpg)
Corners in Images#28
![Page 29: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/29.jpg)
The Aperture Problem#29
![Page 30: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/30.jpg)
#30
Corner Detection
![Page 31: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/31.jpg)
#31
Corner Detection-Basic Idea
![Page 32: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/32.jpg)
Corner Detector: Basic Idea
“flat” region:no change in all directions
“edge”:no change along the edge direction
“corner”:significant change in all directions
![Page 33: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/33.jpg)
Corner Detection Using Intensity: Basic Idea
• Image gradient has two or more dominant directions near a corner.
• Shifting a window in any direction should give a large change in intensity.
“edge”: no change along the edge direction
“corner”: significant change in all directions
“flat” region:no change in all directions
![Page 34: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/34.jpg)
Corner Detection Using Edge Detection?
• Edge detectors are not stable at corners.• Gradient is ambiguous at corner tip.• Discontinuity of gradient direction near
corner.
![Page 35: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/35.jpg)
Corner Types
Example of L-junction, Y-junction, T-junction, Arrow-junction, and X-junction corner types
![Page 36: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/36.jpg)
Mains Steps in Corner Detection
1. For each pixel in the input image, the corner operator is appliedto obtain a cornerness measure for this pixel.
2. Threshold cornerness map to eliminate weak corners.3. Apply non-maximal suppression to eliminate points whose
cornerness measure is not larger than the cornerness values of all points within a certain distance.
![Page 37: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/37.jpg)
Mains Steps in Corner Detection (cont’d)
![Page 38: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/38.jpg)
Moravec Detector (1977)• Measure intensity variation at (x,y) by shifting a
small window (3x3 or 5x5) by one pixel in each of the eight principle directions (horizontally, vertically, and four diagonals).
![Page 39: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/39.jpg)
Moravec Detector (1977)• Calculate intensity variation by taking the sum of
squares of intensity differences of corresponding pixels in these two windows.
∆x, ∆y in {-1,0,1}
SW(-1,-1), SW(-1,0), ...SW(1,1)
8 directions
![Page 40: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/40.jpg)
Moravec Detector (cont’d)• The “cornerness” of a pixel is the minimum intensity variation
found over the eight shift directions:
Cornerness(x,y) = min{SW(-1,-1), SW(-1,0), ...SW(1,1)}
CornernessMap
(normalized)
Note response to isolated points!
![Page 41: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/41.jpg)
Moravec Detector (cont’d)
• Use Non-maximal suppression will yield the final corners.Process of Zero out all pixels that are not the maximum along the direction of the gradient (look at 1 pixel on each side)
![Page 42: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/42.jpg)
Moravec Detector (cont’d)
• Does a reasonable job infinding the majority of true corners.
• Edge points not in one of the eight principle directions will be assigned a relatively large cornerness value.
![Page 43: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/43.jpg)
Moravec Detector (cont’d)• The response is anisotropic (directionally sensitive)
as the intensity variation is only calculated at a discrete set of shifts (i.e., not rotationally invariant)
![Page 44: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/44.jpg)
Corner Detection: Mathematics
2
,( , ) ( , ) ( , ) ( , )
x yE u v w x y I x u y v I x y
Change in appearance of window w(x,y) for the shift [u,v]:
I(x, y)E(u, v)
E(3,2)
w(x, y)
![Page 45: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/45.jpg)
Corner Detection: Mathematics
2
,( , ) ( , ) ( , ) ( , )
x yE u v w x y I x u y v I x y
I(x, y)E(u, v)
E(0,0)
w(x, y)
Change in appearance of window w(x,y) for the shift [u,v]:
![Page 46: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/46.jpg)
Harris Detector: Mathematics
2
,( , ) ( , ) ( , ) ( , )
x yE u v w x y I x u y v I x y
Change of intensity for the shift [u,v]:
IntensityShifted intensity
Window function
orWindow function w(x,y) =
Gaussian1 in window, 0 outside
![Page 47: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/47.jpg)
#47
Harris Detector
• Improves the Moravec operator by avoiding the use of discrete directions and discrete shifts.
• Uses a Gaussian window instead of a square window.
![Page 48: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/48.jpg)
Harris Detector (cont’d)
• Using first-order Taylor approximation:
Reminder: Taylor expansion ( )
2 1f´( ) f´´( ) f ( )( ) f( ) ( ) ( ) ( ) ( )1! 2! !
nn na a af x a x a x a x a O x
n
![Page 49: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/49.jpg)
Harris Detector (cont’d)
Since
![Page 50: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/50.jpg)
Harris Detector (cont’d)
2 x 2 matrix(auto-correlation or2nd order moment matrix)
2( , )( )i if x yx
2( , )( )i if x yy
AW(x,y)=
![Page 51: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/51.jpg)
Harris Detector
2
,( , ) ( , ) ( , ) ( , )
i i
W i i i i i ix y
S x y w x y f x y f x x y y
default window function w(x,y) :
1 in window, 0 outside
• General case – use window function:
22
, ,2
,2
, ,
( , ) ( , )( , )
( , ) ( , )
x x yx y x y x x y
Wx y x y y
x y yx y x y
w x y f w x y f ff f f
A w x yf f f
w x y f f w x y f
![Page 52: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/52.jpg)
Harris Detector (cont’d)
window function w(x,y) :Gaussian
• Harris uses a Gaussian window: w(x,y)=G(x,y,σI) where σIis called the “integration” scale
22
, ,2
,2
, ,
( , ) ( , )( , )
( , ) ( , )
x x yx y x y x x y
Wx y x y y
x y yx y x y
w x y f w x y f ff f f
A w x yf f f
w x y f f w x y f
![Page 53: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/53.jpg)
Harris Detector
Describes the gradientdistribution (i.e., local structure)inside window!
Does not depend on
2
2,
x x yW
x y x y y
f f fA
f f f
,x y
![Page 54: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/54.jpg)
Harris Detector (cont’d)
Since M is symmetric, we have: 11
2
00WA R R
We can visualize AW as an ellipse with axis lengths determined by the eigenvalues and orientation determined by R
(λmin)-1/2
(λmax)-1/2[ ] constW
xx y A
y
Ellipse equation:direction of the slowest change
direction of the fastest change
![Page 55: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/55.jpg)
Harris Corner Detector (cont’d)
• The eigenvectors of AW encode direction of intensity change.
• The eigenvalues of AW encode strength of intensity change.
v1
v2
![Page 56: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/56.jpg)
Harris Detector (cont’d)
• Eigenvectors encode edge direction
• Eigenvalues encode edge strength
(λmin)-1/2
(λmax)-1/2
direction of the fastest change
direction of the slowest change
![Page 57: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/57.jpg)
Distribution of fx and fy
![Page 58: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/58.jpg)
Distribution of fx and fy (cont’d)
![Page 59: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/59.jpg)
Harris Detector (cont’d)
(assuming that λ1 > λ2)
2
%
![Page 60: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/60.jpg)
Harris Detector
Measure of corner response:
(k – empirical constant, k = 0.04-0.06)
(Shi-Tomasi variation: use min(λ1,λ2) instead of R)
http
://w
ww
.wis
dom
.wei
zman
n.ac
.il/~
deni
ss/v
isio
n_sp
ring0
4/fil
es/In
varia
ntFe
atur
es.p
ptD
arya
Fro
lova
, Den
is S
imak
ov T
he W
eizm
ann
Inst
itute
of S
cien
ce
![Page 61: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/61.jpg)
Harris Detector: Mathematics
“Corner”
“Edge”
“Edge”
“Flat”
• R depends only on eigenvalues of M
• R is large for a corner
• R is negative with large magnitude for an edge
• |R| is small for a flatregion
R > 0
R < 0
R < 0|R| small
![Page 62: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/62.jpg)
Harris Corner Detector (cont’d)
1
2
“Corner”1 and 2 are large,1 ~ 2;intensity changes in all directions
1 and 2 are small;intensity is almost constant in all directions
“Edge” 1 >> 2
“Edge” 2 >> 1
“Flat” region
Classification of pixels using the eigenvalues of AW :
![Page 63: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/63.jpg)
Harris Detector (cont’d)
( , ) ( , , )* ( , )i iD i i
f x y G x y f x yx x
( , ) ( , , )* ( , )i iD i i
f x y G x y f x yy y
σD is called the “differentiation” scale
![Page 64: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/64.jpg)
Harris Detector
• The Algorithm:– Find points with large corner response function
R (R > threshold)– Take the points of local maxima of R
![Page 65: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/65.jpg)
65
Harris corner detector algorithm• Compute image gradients Ix Iy for all pixels• For each pixel
– Compute
by looping over neighbors x,y– compute
• Find points with large corner response function R(R > threshold)
• Take the points of locally maximum R as the detected feature points (ie, pixels where R is bigger than for all the 4 or 8 neighbors).
65
http
://w
ww
.wis
dom
.wei
zman
n.ac
.il/~
deni
ss/v
isio
n_sp
ring0
4/fil
es/In
varia
ntFe
atur
es.p
ptD
arya
Fro
lova
, Den
is S
imak
ov T
he W
eizm
ann
Inst
itute
of S
cien
ce
![Page 66: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/66.jpg)
Harris Corner Detector (cont’d)• To avoid computing the eigenvalues
explicitly, the Harris detector uses the following function:
R(AW) = det(AW) – α trace2(AW)
which is equal to:
R(AW) = λ1 λ2- α (λ1+ λ2)2
α: is a const
![Page 67: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/67.jpg)
Harris Corner Detector (cont’d)
“Corner”R > 0
“Edge” R < 0
“Edge” R < 0
“Flat” region
|R| smallα: is usually between 0.04 and 0.06
R(AW) = det(AW) – α trace2(AW)
Classification of image
points using R(AW):
![Page 68: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/68.jpg)
Harris Corner Detector (cont’d)
• Other functions:
2 1
1 2
1 2
det
a
AtrA
![Page 69: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/69.jpg)
Harris Corner Detector - Steps
( , , )* ( , )x D i if G x y f x yx
( , , )* ( , )y D i if G x y f x yy
σD is called the “differentiation” scale
1. Compute the horizontal and vertical (Gaussian) derivatives
2. Compute the three images involved in AW :
2
2,
( , ) x x yW
x W y W x y y
f f fA w x y
f f f
![Page 70: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/70.jpg)
Harris Detector - Steps
R(AW) = det(AW) – α trace2(AW)
3. Convolve each of the three images with a larger Gaussian
w(x,y) :Gaussian
σI is called the “integration” scale
4. Determine cornerness:
5. Find local maxima
![Page 71: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/71.jpg)
Harris Detector - Example
![Page 72: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/72.jpg)
Harris Detector - Example
![Page 73: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/73.jpg)
Harris Detector - ExampleCompute corner response R
![Page 74: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/74.jpg)
Harris Detector - ExampleFind points with large corner response: R>threshold
![Page 75: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/75.jpg)
Harris Detector - ExampleTake only the points of local maxima of R
![Page 76: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/76.jpg)
Harris Detector - ExampleMap corners on the original image
![Page 77: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/77.jpg)
Harris Detector – Scale Parameters
• The Harris detector requires two scale parameters:
(i) a differentiation scale σD for smoothing prior to the computation of image derivatives,
&(ii) an integration scale σI for defining the size of the Gaussian window (i.e., integrating derivative responses). AW(x,y) AW(x,y,σI,σD)
• Typically, σI=γσD
![Page 78: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/78.jpg)
Invariance to Geometric/Photometric Changes
• Is the Harris detector invariant to geometric and photometric changes?
• Geometric
– Rotation
– Scale
– Affine
• Photometric– Affine intensity change: I(x,y) a I(x,y) + b
![Page 79: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/79.jpg)
Harris Detector: Rotation Invariance
• Rotation
Ellipse rotates but its shape (i.e. eigenvalues) remains the same
Corner response R is invariant to image rotation
![Page 80: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/80.jpg)
Harris Detector: Rotation Invariance (cont’d)
![Page 81: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/81.jpg)
Harris Detector: Photometric Changes
• Affine intensity change Only derivatives are used => invariance to intensity shift I(x,y) I (x,y) + b
Intensity scale: I(x,y) a I(x,y)
R
x (image coordinate)
threshold
R
x (image coordinate)
Partially invariant to affine intensity change
![Page 82: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/82.jpg)
Harris Detector: Scale Invariance
• Scaling
All points will be classified as edges
Corner
Not invariant to scaling (and affine transforms)
![Page 83: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/83.jpg)
Harris Detector: Repeatability
• In a comparative study of different interest point detectors, Harris was shown to be the most repeatable and most informative.
C. Schmid, R. Mohr, and C. Bauckhage, "Evaluation of Interest Point Detectors", International Journal of Computer Vision 37(2), 151.172, 2000.
best results:σI=1, σD=2
Repeatability = 1 2
#min( , )
correspondencesm m
![Page 84: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/84.jpg)
Harris Detector: Disadvantages
• Sensitive to:– Scale change– Significant viewpoint change– Significant contrast change
![Page 85: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/85.jpg)
How to handle scale changes?
• AW must be adapted to scale changes.
• If the scale change is known, we can adapt the Harris detector to the scale change (i.e., set properly σI ,σD).
• What if the scale change is unknown?
![Page 86: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/86.jpg)
Multi-scale Harris Detector
scale
x
y
Harris
• Detects interest points at varying scales.R(AW) = det(AW(x,y,σI,σD)) – α trace2(AW(x,y,σI,σD))
σnσD= σnσI=γσD
σn=knσ
![Page 87: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/87.jpg)
How to cope with transformations?
• Exhaustive search• Invariance• Robustness
![Page 88: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/88.jpg)
Exhaustive search• Multi-scale approach
Slide from T. Tuytelaars ECCV 2006 tutorial
![Page 89: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/89.jpg)
Exhaustive search• Multi-scale approach
![Page 90: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/90.jpg)
Exhaustive search
• Multi-scale approach
![Page 91: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/91.jpg)
Exhaustive search• Multi-scale approach
![Page 92: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/92.jpg)
How to handle scale changes?
• Not a good idea!– There will be many points representing the same
structure, complicating matching! – Note that point locations shift as scale increases.
The size of the circle corresponds to the scale at which the point was detected
![Page 93: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/93.jpg)
How to handle scale changes? (cont’d)
• How do we choose corresponding circles independently in each image?
![Page 94: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/94.jpg)
How to handle scale changes? (cont’d)
• Alternatively, use scale selection to find the characteristic scale of each feature.
• Characteristic scale depends on the feature’s spatial extent (i.e., local neighborhood of pixels).
scale selection scale selection
![Page 95: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/95.jpg)
How to handle scale changes?
The size of the circles corresponds to the scale at which the point was selected.
• Only a subset of the points computed in scale space are selected!
![Page 96: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/96.jpg)
Automatic Scale Selection• Design a function F(x,σn) which provides some local
measure. • Select points at which F(x,σn) is maximal over σn.
T. Lindeberg, "Feature detection with automatic scale selection" International Journal of Computer Vision, vol. 30, no. 2, pp 77-116, 1998.
max of F(x,σn)corresponds tocharacteristic scale!
σn
F(x,σn)
![Page 97: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/97.jpg)
Invariance
• Extract patch from each image individually
![Page 98: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/98.jpg)
Automatic scale selection• Solution:
– Design a function on the region, which is “scale invariant” (the same for corresponding regions, even if they are at different scales)
Example: average intensity. For corresponding regions (even of different sizes) it will be the same.
scale = 1/2
– For a point in one image, we can consider it as a function of region size (patch width)
f
region size
Image 1 f
region size
Image 2
![Page 99: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/99.jpg)
Automatic scale selection• Common approach:
scale = 1/2f
region size
Image 1 f
region size
Image 2
Take a local maximum of this function
Observation: region size, for which the maximum is achieved, should be invariant to image scale.
s1 s2
Important: this scale invariant region size is found in each image independently!
![Page 100: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/100.jpg)
Automatic Scale Selection
100
)),(( )),((11
xIfxIfmm iiii
Same operator responses if the patch contains the same image up to scale factor.
![Page 101: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/101.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
101)),((
1xIf
mii )),((1
xIfmii
![Page 102: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/102.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
102)),((
1xIf
mii )),((1
xIfmii
![Page 103: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/103.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
103)),((
1xIf
mii )),((1
xIfmii
![Page 104: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/104.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
104)),((
1xIf
mii )),((1
xIfmii
![Page 105: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/105.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
105)),((
1xIf
mii )),((1
xIfmii
![Page 106: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/106.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
106)),((
1xIf
mii )),((1
xIfmii
![Page 107: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/107.jpg)
Scale selection• Use the scale determined by detector to compute
descriptor in a normalized frame
![Page 108: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/108.jpg)
Automatic Scale Selection (cont’d)
• Characteristic scale is relatively independentof the image scale.
• The ratio of the scale values corresponding to the max values, is equal to the scale factor between the images.
Scale selection allows for finding spatial extend that is covariant with the image transformation.
Scale factor: 2.5
![Page 109: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/109.jpg)
Automatic Scale Selection
• What local measures should we use? – Should be rotation invariant– Should have one stable sharp peak
![Page 110: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/110.jpg)
How should we choose F(x,σn) ?
• Typically, F(x,σn) is defined using derivatives, e.g.:
2 2 2
2
12
: ( ( , ) ( , ))
: | ( ( , ) ( , )) |
:| ( )* ( ) ( )* ( ) |
: det( ) ( )
x y
xx yy
n n
W W
Square gradient L x L x
LoG L x L x
DoG I x G I x G
Harris function A trace A
C. Schmid, R. Mohr, and C. Bauckhage, "Evaluation of Interest Point Detectors", International Journal of Computer Vision, 37(2), pp. 151-172, 2000.
![Page 111: Lecture 8: Interest Point Detection - University of Minnesotadept.me.umn.edu/courses/me5286/vision/VisionNotes/...Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu](https://reader034.fdocuments.net/reader034/viewer/2022042400/5f0f622f7e708231d443e3b7/html5/thumbnails/111.jpg)
Review#111• Patch/Template Matching
• Interest Points• Corner Detection: Harris Detector
• Properties of Harris Detector• Scaling approach