Local Image Features
description
Transcript of Local Image Features
![Page 1: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/1.jpg)
Local Image Features
Computer VisionCS 143, Brown
James Hays
Acknowledgment: Many slides from Derek Hoiem and Grauman&Leibe 2008 AAAI Tutorial
Read Szeliski 4.1
![Page 2: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/2.jpg)
This section: correspondence and alignment
• Correspondence: matching points, patches, edges, or regions across images
≈
![Page 3: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/3.jpg)
Overview of Keypoint Matching
K. Grauman, B. Leibe
Af Bf
B1
B2
B3A1
A2 A3
Tffd BA ),(
1. Find a set of distinctive key- points
3. Extract and normalize the region content
2. Define a region around each keypoint
4. Compute a local descriptor from the normalized region
5. Match local descriptors
![Page 4: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/4.jpg)
Review: Harris corner detector• Approximate distinctiveness by local auto-
correlation.• Approximate local auto-correlation by
second moment matrix• Quantify distinctiveness (or cornerness) as
function of the eigenvalues of the second moment matrix.
• But we don’t actually need to compute the eigenvalues byusing the determinant and traceof the second moment matrix.
E(u, v)
(max)-1/2
(min)-1/2
![Page 5: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/5.jpg)
Harris Detector [Harris88]
• Second moment matrix
)()()()(
)(),( 2
2
DyDyx
DyxDxIDI III
IIIg
5
1. Image derivatives
2. Square of derivatives
3. Gaussian filter g(I)
Ix Iy
Ix2 Iy2 IxIy
g(Ix2) g(Iy2) g(IxIy)
222222 )]()([)]([)()( yxyxyx IgIgIIgIgIg
])),([trace()],(det[ 2DIDIhar
4. Cornerness function – both eigenvalues are strong
har5. Non-maxima suppression
1 2
1 2
dettrace
MM
(optionally, blur first)
![Page 6: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/6.jpg)
So far: can localize in x-y, but not scale
![Page 7: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/7.jpg)
Automatic Scale Selection
K. Grauman, B. Leibe
)),(( )),((11
xIfxIfmm iiii
How to find corresponding patch sizes?
![Page 8: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/8.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
![Page 9: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/9.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
![Page 10: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/10.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
![Page 11: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/11.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
![Page 12: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/12.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
![Page 13: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/13.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
![Page 14: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/14.jpg)
What Is A Useful Signature Function?• Difference-of-Gaussian = “blob” detector
K. Grauman, B. Leibe
![Page 15: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/15.jpg)
Difference-of-Gaussian (DoG)
K. Grauman, B. Leibe
- =
![Page 16: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/16.jpg)
DoG – Efficient Computation• Computation in Gaussian scale pyramid
K. Grauman, B. Leibe
Original image 41
2
Sampling withstep 4 =2
![Page 17: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/17.jpg)
Find local maxima in position-scale space of Difference-of-Gaussian
K. Grauman, B. Leibe
)()( yyxx LL
2
3
4
5
List of (x, y, s)
![Page 18: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/18.jpg)
Results: Difference-of-Gaussian
K. Grauman, B. Leibe
![Page 19: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/19.jpg)
T. Tuytelaars, B. Leibe
Orientation Normalization• Compute orientation histogram• Select dominant orientation• Normalize: rotate to fixed orientation
0 2p
[Lowe, SIFT, 1999]
![Page 20: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/20.jpg)
Maximally Stable Extremal Regions [Matas ‘02]
• Based on Watershed segmentation algorithm• Select regions that stay stable over a large
parameter range
K. Grauman, B. Leibe
![Page 21: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/21.jpg)
Example Results: MSER
23 K. Grauman, B. Leibe
![Page 22: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/22.jpg)
Image representations
• Templates– Intensity, gradients, etc.
• Histograms– Color, texture, SIFT descriptors, etc.
![Page 23: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/23.jpg)
Space Shuttle Cargo Bay
Image Representations: Histograms
Global histogram• Represent distribution of features
– Color, texture, depth, …
Images from Dave Kauchak
![Page 24: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/24.jpg)
Image Representations: Histograms
• Joint histogram– Requires lots of data– Loss of resolution to
avoid empty binsImages from Dave Kauchak
Marginal histogram• Requires independent features• More data/bin than
joint histogram
Histogram: Probability or count of data in each bin
![Page 25: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/25.jpg)
What kind of things do we compute histograms of?
• Color
• Texture (filter banks or HOG over regions)
L*a*b* color space HSV color space
![Page 26: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/26.jpg)
What kind of things do we compute histograms of?
• Histograms of oriented gradients
SIFT – Lowe IJCV 2004
![Page 27: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/27.jpg)
SIFT vector formation• Computed on rotated and scaled version of window
according to computed orientation & scale– resample the window
• Based on gradients weighted by a Gaussian of variance half the window (for smooth falloff)
![Page 28: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/28.jpg)
SIFT vector formation• 4x4 array of gradient orientation histogram weighted
by magnitude• 8 orientations x 4x4 array = 128 dimensions• Motivation: some sensitivity to spatial layout, but not
too much.
showing only 2x2 here but is 4x4
![Page 29: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/29.jpg)
Ensure smoothness• Gaussian weight • Trilinear interpolation
– a given gradient contributes to 8 bins: 4 in space times 2 in orientation
![Page 30: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/30.jpg)
Reduce effect of illumination• 128-dim vector normalized to 1 • Threshold gradient magnitudes to avoid excessive
influence of high gradients– after normalization, clamp gradients >0.2– renormalize
![Page 31: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/31.jpg)
Local Descriptors: Shape Context
Count the number of points inside each bin, e.g.:
Count = 4
Count = 10...
Log-polar binning: more precision for nearby points, more flexibility for farther points.
Belongie & Malik, ICCV 2001K. Grauman, B. Leibe
![Page 32: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/32.jpg)
Shape Context Descriptor
![Page 33: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/33.jpg)
Local Descriptors: Geometric Blur
Example descriptor
~
Compute edges at four orientations
Extract a patchin each channel
Apply spatially varyingblur and sub-sample
(Idealized signal)
Berg & Malik, CVPR 2001K. Grauman, B. Leibe
![Page 34: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/34.jpg)
Self-similarity Descriptor
Matching Local Self-Similarities across Images and Videos, Shechtman and Irani, 2007
![Page 35: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/35.jpg)
Self-similarity Descriptor
Matching Local Self-Similarities across Images and Videos, Shechtman and Irani, 2007
![Page 36: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/36.jpg)
Self-similarity Descriptor
Matching Local Self-Similarities across Images and Videos, Shechtman and Irani, 2007
![Page 37: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/37.jpg)
Learning Local Image Descriptors, Winder and Brown, 2007
![Page 38: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/38.jpg)
Local Descriptors• Most features can be thought of as templates,
histograms (counts), or combinations• The ideal descriptor should be
– Robust– Distinctive– Compact– Efficient
• Most available descriptors focus on edge/gradient information– Capture texture information– Color rarely used
K. Grauman, B. Leibe
![Page 39: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/39.jpg)
Matching Local Features• Nearest neighbor (Euclidean distance)• Threshold ratio of nearest to 2nd nearest descriptor
Lowe IJCV 2004
![Page 40: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/40.jpg)
SIFT Repeatability
Lowe IJCV 2004
![Page 41: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/41.jpg)
SIFT Repeatability
![Page 42: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/42.jpg)
SIFT Repeatability
Lowe IJCV 2004
![Page 43: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/43.jpg)
Local Descriptors: SURF
K. Grauman, B. Leibe
• Fast approximation of SIFT idea Efficient computation by 2D box filters &
integral images 6 times faster than SIFT
Equivalent quality for object identification
[Bay, ECCV’06], [Cornelis, CVGPU’08]
• GPU implementation available Feature extraction @ 200Hz
(detector + descriptor, 640×480 img) http://www.vision.ee.ethz.ch/~surf
![Page 44: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/44.jpg)
Local Descriptors: Shape Context
Count the number of points inside each bin, e.g.:
Count = 4
Count = 10...
Log-polar binning: more precision for nearby points, more flexibility for farther points.
Belongie & Malik, ICCV 2001K. Grauman, B. Leibe
![Page 45: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/45.jpg)
Local Descriptors: Geometric Blur
Example descriptor
~
Compute edges at four orientations
Extract a patchin each channel
Apply spatially varyingblur and sub-sample
(Idealized signal)
Berg & Malik, CVPR 2001K. Grauman, B. Leibe
![Page 46: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/46.jpg)
Choosing a detector
• What do you want it for?– Precise localization in x-y: Harris– Good localization in scale: Difference of Gaussian– Flexible region shape: MSER
• Best choice often application dependent– Harris-/Hessian-Laplace/DoG work well for many natural categories– MSER works well for buildings and printed things
• Why choose?– Get more points with more detectors
• There have been extensive evaluations/comparisons– [Mikolajczyk et al., IJCV’05, PAMI’05]– All detectors/descriptors shown here work well
![Page 47: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/47.jpg)
Comparison of Keypoint Detectors
Tuytelaars Mikolajczyk 2008
![Page 48: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/48.jpg)
Choosing a descriptor
• Again, need not stick to one
• For object instance recognition or stitching, SIFT or variant is a good choice
![Page 49: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/49.jpg)
Things to remember
• Keypoint detection: repeatable and distinctive– Corners, blobs, stable regions– Harris, DoG
• Descriptors: robust and selective– spatial histograms of orientation– SIFT
![Page 50: Local Image Features](https://reader035.fdocuments.net/reader035/viewer/2022062814/568167be550346895ddd106a/html5/thumbnails/50.jpg)
Next time
• Feature tracking