Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching...
Transcript of Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching...
![Page 1: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/1.jpg)
1 Multi-resolution image recognition
Multi-resolution image recognition
Jean-Baptiste Boin Roland Angst David Chen Bernd Girod
![Page 2: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/2.jpg)
2 Multi-resolution image recognition
Outline • Scale distribution • Presentation of two different approaches and
experiments • Analysis of previous results
![Page 3: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/3.jpg)
3 Multi-resolution image recognition
Motivation
• Typical image retrieval applications: similar resolution in database images and queries
• Performance drops when the resolutions are very different (high-res database image vs. low-res query)
• OK for some applications (product recognition), not ideal for others (large painting recognition)
Full view Detail
![Page 4: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/4.jpg)
4 Multi-resolution image recognition
Scale distribution – derivation
• Goal: Find the average distribution of scales for an “ideal” feature detector
• Hypotheses: continuous representation of the scales + a few assumptions on the feature detectors
![Page 5: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/5.jpg)
5 Multi-resolution image recognition
Scale distribution – derivation
• F(2s�) = 0.75 – Qualitative justification
F(s) = 0 if s ≤ s� 1 - s�²/s² if s ≥ s�
ρ(s) � 1/s³
N features
N/4 features 50% downsampling
Correspond to features of scale s ≥ 2s� on original image {s� ≤ s ≤ 2s�} contains N-N/4 = 3N/4 features
![Page 6: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/6.jpg)
6 Multi-resolution image recognition
Reference images
Queries (2x)
• Images extracted from a public art repository (Web Gallery of Art): more than 30,000 images
• We keep cropped regions of fixed size (17,146 images at resolution 1024x768)
• We generate queries of half the size of the database images (512x384) by rotating/scaling/translating
Queries (1x)
2x
1x
Dataset used for the experiments
(zoom factor defined before downsampling)
![Page 7: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/7.jpg)
7 Multi-resolution image recognition
Queries (2x)
Scale distribution – experiment
Reference images
SURF features
SURF features
Match + RANSAC
all extracted features
matching features
![Page 8: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/8.jpg)
8 Multi-resolution image recognition
Scale distribution – results
Effect of discretized scale detector
![Page 9: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/9.jpg)
9 Multi-resolution image recognition
Scale distribution – results
• Power law: a bit of a stretch, but gives a rough idea of the behavior
![Page 10: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/10.jpg)
10 Multi-resolution image recognition
Baseline (single REVV)
• Aggregate 250 SURF features with coarsest scale
SURF
coarse
fine
REVV Queries
Reference images
SURF
coarse
fine
REVV Get rank
N REVV comparisons
![Page 11: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/11.jpg)
11 Multi-resolution image recognition
Tile based approach
• Each DB image is represented by 5 tiles
SURF
coarse
fine
REVV Queries
Reference images (5 tiles per orig. image)
SURF
coarse
fine
REVV Get rank (best tile)
w
h
h
w
5N REVV comparisons
![Page 12: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/12.jpg)
12 Multi-resolution image recognition
Scale based aggregation – Idea
• Main conclusion from previous analysis: most features have a scale in the interval [s0, 2s0] – 75% in theory – ~70% in practice (depends of size of query)
• Aggregate features according to scale domains and position
s > s1
s2 < s < s1
…
s2 < s < s1 s2 < s < s1
s2 < s < s1
![Page 13: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/13.jpg)
13 Multi-resolution image recognition
Scale based aggregation – Idea
…
s > s1 s2 < s < s1 s3 < s < s2
s2 s3 s1
query query query
![Page 14: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/14.jpg)
14 Multi-resolution image recognition
Scale based aggregation – Idea
…
s > s1 s2 < s < s1 s3 < s < s2
s2 s3 s1
query query query
![Page 15: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/15.jpg)
15 Multi-resolution image recognition
Scale based aggregation – Experiment
• Database side: case of limited scale variation, we only consider 2 levels
• Query side: we only have 2 bins (s > 2s0 and s < 2s0)
s > 2s0
s < 2s0 s < 2s0
s < 2s0 s < 2s0
![Page 16: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/16.jpg)
16 Multi-resolution image recognition
Scale based aggregation – Experiment
SURF
Queries
SURF
REVV (s ≥ s�)
REVV (s ≤ s�)
REVV (s ≤ s�)
REVV (s ≤ s�)
REVV (s ≤ s�)
REVV (s ≥ s�)
REVV (s ≤ s�)
Reference images
![Page 17: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/17.jpg)
17 Multi-resolution image recognition
Scale based aggregation
• How do we merge the two lists? 1. By cheating: we take the “best” rank in each list 2. By using the correlation scores to re-rank the results 3. By using a linear combination of the best correlation score
for each image
correlation (fine list)
correlation (coarse list)
![Page 18: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/18.jpg)
18 Multi-resolution image recognition
Multi-scale experiments
• Zoom = 2x (query represents ~25% of original image)
![Page 19: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/19.jpg)
19 Multi-resolution image recognition
Multi-scale experiments
• Zoom = 1.5x (query represents ~44% of original image)
![Page 20: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/20.jpg)
20 Multi-resolution image recognition
Multi-scale experiments
• Zoom = 1x (query represents ~100% of original image)
![Page 21: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/21.jpg)
21 Multi-resolution image recognition
Analysis of results
• Current problem of our approach: hard-binned scale (assumes good reproducibility of scale extraction)
• Justification of the good results obtained in the tiling
approach: REVV and surface overlap
![Page 22: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/22.jpg)
22 Multi-resolution image recognition
Queries (2x)
Scale reproducibility – experiment
Reference images
SURF features
SURF features
Match + RANSAC
log scale of matching features in both images
![Page 23: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/23.jpg)
23 Multi-resolution image recognition
Scale reproducibility – results
additive noise
![Page 24: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/24.jpg)
24 Multi-resolution image recognition
Scale reproducibility – results
additive noise
![Page 25: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/25.jpg)
25 Multi-resolution image recognition
Queries (2x)
REVV and surface overlap – experiment
Reference images
SURF features
SURF features
log rank (for each type of tile)
Queries
REVV
REVV
Upper-left tile Upper-right tile
Lower-left tile Lower-right tile
Downsamp. tile
![Page 26: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/26.jpg)
26 Multi-resolution image recognition
REVV and surface overlap – results Precision at rank 10 for each type of tile
Strongly correlated with area overlap
50% overlap: ~90% precision at rank 10
25% overlap: ~70% precision at rank 10
![Page 27: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/27.jpg)
27 Multi-resolution image recognition
Conclusion of multi-resolution exploration
• Considerable unsolved issues – Scale reproducibility (try other values of thresholds) – Increased cost of running 2 queries, but no real gain in non-
optimal conditions
• The simpler (tile-based) approach is “too good” – Shows the robustness of REVV
• Hybrid approach?
![Page 28: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/28.jpg)
28 Multi-resolution image recognition
Conclusion of multi-resolution exploration
• Hybrid approach
2s0 s0
s < 2s0 s < 2s0
s < 2s0 s < 2s0 s > 2s0 s < 2s0
query query
s > 2s0
![Page 29: Multi-resolution image recognition · SURF features Match + RANSAC all extracted features matching features . Multi-resolution image recognition 8 Scale distribution – results Effect](https://reader034.fdocuments.net/reader034/viewer/2022050516/5fa0050a4df0da0ca36bd4e1/html5/thumbnails/29.jpg)
29 Multi-resolution image recognition
Conclusion of multi-resolution exploration
• Hybrid approach
2s0 s0
s < 2s0 s < 2s0
s < 2s0 s < 2s0 s < 2s0
query
s > 2s0 s > 2s0
s > 2s0 s > 2s0
query s > 2s0
drawback: doubles the storage requirement
possibility to reduce computation amount