Laurent Lefèvre INRIA / LIP (UMR CNRS, INRIA, ENS, UCB) [email protected]
Overview - Inria
Transcript of Overview - Inria
![Page 1: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/1.jpg)
Overview
• Introduction to local features
• Harris interest points + SSD, ZNCC, SIFT
• Scale & affine invariant interest point detectors• Scale & affine invariant interest point detectors
• Evaluation and comparison of different detectors
• Region descriptors and their performance
![Page 2: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/2.jpg)
Affine invariant regions - Motivation
• Scale invariance is not sufficient for large baseline changes
A
detected scale invariant region
A
projected regions, viewpoint changes can locally be approximated by an affine transformation A
![Page 3: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/3.jpg)
Affine invariant regions - Motivation
![Page 4: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/4.jpg)
Affine invariant regions - Example
![Page 5: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/5.jpg)
Harris/Hessian/Laplacian-Affine
• Initialize with scale-invariant Harris/Hessian/Laplacian points
• Estimation of the affine neighbourhood with the second moment matrix [Lindeberg’94]
• Apply affine neighbourhood estimation to the scale-invariant interest points [Mikolajczyk & Schmid’02, Schaffalitzky & Zisserman’02]
• Excellent results in a comparison [Mikolajczyk et al.’05]
![Page 6: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/6.jpg)
Affine invariant regions
• Based on the second moment matrix (Lindeberg’94)
⊗==
),(),(
),(),()(),,( 2
2
2
DyDyx
DyxDx
IDDI LLL
LLLGM
σσσσ
σσσσµxx
xxx
xx 2
1
M=′
• Normalization with eigenvalues/eigenvectors
![Page 7: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/7.jpg)
Affine invariant regions
LR xx A=
′=′LR Rxx
Isotropic neighborhoods related by image rotation
L2
1
L xx LM=′R
2
1
R xx RM=′
![Page 8: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/8.jpg)
• Iterative estimation – initial points
Affine invariant regions - Estimation
![Page 9: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/9.jpg)
• Iterative estimation – iteration #1
Affine invariant regions - Estimation
![Page 10: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/10.jpg)
• Iterative estimation – iteration #2
Affine invariant regions - Estimation
![Page 11: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/11.jpg)
• Iterative estimation – iteration #3, #4
Affine invariant regions - Estimation
![Page 12: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/12.jpg)
Harris-Affine versus Harris-Laplace
Harris-LaplaceHarris-Affine
![Page 13: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/13.jpg)
Harris-Affine
Harris/Hessian-Affine
Hessian-Affine
![Page 14: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/14.jpg)
Harris-Affine
![Page 15: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/15.jpg)
Hessian-Affine
![Page 16: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/16.jpg)
Matches
22 correct matches
![Page 17: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/17.jpg)
Matches
33 correct matches
![Page 18: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/18.jpg)
Maximally stable extremal regions (MSER) [Matas’02]
• Based on the idea of regionsegmentation
• State of the art results
![Page 19: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/19.jpg)
Maximally stable extremal regions (MSER) [Matas’02]
• Extremal regions: connected components in a thresholded image (all pixels above/below a threshold)
• Maximally stable: minimal change of the component (area) for a change of the threshold, i.e. region remains (area) for a change of the threshold, i.e. region remains stable for a change of threshold
![Page 20: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/20.jpg)
Maximally stable extremal regions (MSER)
Examples of thresholded images
high threshold
low threshold
![Page 21: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/21.jpg)
MSER
![Page 22: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/22.jpg)
Overview
• Introduction to local features
• Harris interest points + SSD, ZNCC, SIFT
• Scale & affine invariant interest point detectors• Scale & affine invariant interest point detectors
• Evaluation and comparison of different detectors
• Region descriptors and their performance
![Page 23: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/23.jpg)
Evaluation of interest points
• Quantitative evaluation of interest point/region detectors– points / regions at the same relative location and area
• Repeatability rate : percentage of corresponding points
• Two points/regions are corresponding if– location error small– area intersection large
• [K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky, T. Kadir & L. Van Gool ’05]
![Page 24: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/24.jpg)
Evaluation criterion
H
%100#
# ⋅=regionsdetected
regionsingcorrespondityrepeatabil
![Page 25: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/25.jpg)
Evaluation criterion
H
%100#
# ⋅=regionsdetected
regionsingcorrespondityrepeatabil
%100)1( ⋅−=union
onintersectierroroverlap
2% 10% 20% 30% 40% 50% 60%
![Page 26: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/26.jpg)
Comparison of affine invariant detectors
60
70
80
90
100
repe
atab
ility
%
Harris−Affine
Hessian−Affine
MSER
IBR
EBR
Salient
Viewpoint change - structured scene
repeatability %
15 20 25 30 35 40 45 50 55 60 650
10
20
30
40
50
60
viewpoint angle
repe
atab
ility
%
reference image 20 6040
![Page 27: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/27.jpg)
Scale change – textured scene
repeatability %
Comparison of affine invariant detectors
reference image 4
![Page 28: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/28.jpg)
• Good performance for large viewpoint and scale changes
• Results depend on transformation and scene type, no one best detector
Conclusion - detectors
• Detectors are complementary– MSER adapted to structured scenes– Harris and Hessian adapted to textured scenes
• Performance of the different scale invariant detectors is very similar (Harris-Laplace, Hessian, LoG and DOG)
• Scale-invariant detector sufficient up to 40 degrees of viewpoint change
![Page 29: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/29.jpg)
Overview
• Introduction to local features
• Harris interest points + SSD, ZNCC, SIFT
• Scale & affine invariant interest point detectors• Scale & affine invariant interest point detectors
• Evaluation and comparison of different detectors
• Region descriptors and their performance
![Page 30: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/30.jpg)
Region descriptors
• Normalized regions are– invariant to geometric transformations except rotation– not invariant to photometric transformations
![Page 31: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/31.jpg)
Descriptors
• Regions invariant to geometric transformations except rotation– rotation invariant descriptors– normalization with dominant gradient direction– normalization with dominant gradient direction
• Regions not invariant to photometric transformations– invariance to affine photometric transformations– normalization with mean and standard deviation of the image patch
![Page 32: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/32.jpg)
Descriptors
Extract affine regions Normalize regionsEliminate rotational
+ illumination
Compute appearancedescriptors
SIFT (Lowe ’04)
![Page 33: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/33.jpg)
Descriptors
• Gaussian derivative-based descriptors– Differential invariants (Koenderink and van Doorn’87)– Steerable filters (Freeman and Adelson’91)
• SIFT (Lowe’99)• SIFT (Lowe’99)• Moment invariants [Van Gool et al.’96]
• Shape context [Belongie et al.’02]
• SIFT with PCA dimensionality reduction• Gradient PCA [Ke and Sukthankar’04]
• SURF descriptor [Bay et al.’08]
• DAISY descriptor [Tola et al.’08, Windler et al’09]
![Page 34: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/34.jpg)
Comparison criterion
• Descriptors should be– Distinctive– Robust to changes on viewing conditions as well as to errors of
the detector
• Detection rate (recall) 1• Detection rate (recall)– #correct matches / #correspondences
• False positive rate– #false matches / #all matches
• Variation of the distance threshold – distance (d1, d2) < threshold
1
1
[K. Mikolajczyk & C. Schmid, PAMI’05]
![Page 35: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/35.jpg)
Viewpoint change (60 degrees)
0.6
0.7
0.8
0.9
1
#cor
rect
/ 21
01
esift* *shape context
gradient pcacross correlation
complex filtershar−aff esift
steerable filters
gradient moments
sift
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
1−precision
#cor
rect
/ 21
01
![Page 36: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/36.jpg)
esift* *
Scale change (factor 2.8)
0.6
0.7
0.8
0.9
1
#cor
rect
/ 20
86
shape context
gradient pcacross correlation
complex filtershar−aff esift
steerable filters
gradient moments
sift
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
1−precision
#cor
rect
/ 20
86
![Page 37: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/37.jpg)
Conclusion - descriptors
• SIFT based descriptors perform best
• Significant difference between SIFT and low dimension descriptors as well as cross-correlation
• Robust region descriptors better than point-wise descriptors
• Performance of the descriptor is relatively independent of the detector
![Page 38: Overview - Inria](https://reader036.fdocuments.net/reader036/viewer/2022081409/6297f25820c018295c39f607/html5/thumbnails/38.jpg)
Available on the internet
• Binaries for detectors and descriptors– Building blocks for recognition systems
• Carefully designed test setup• Carefully designed test setup– Dataset with transformations– Evaluation code in matlab– Benchmark for new detectors and descriptors
http://lear.inrialpes.fr/software