Post on 18-Aug-2020
ARMIN MUSTAFA, HANSUNG KIM,
EVREN IMRE AND ADRIAN HILTON
Segmentation based features for wide-
baseline multi -view reconstruction
Contents
• Motivation
• Existing methods
• SFD: Segmentation based feature detector
• Results and Evaluation of feature detectors
• Application to sparse and dense reconstruction
• Conclusion
2
Applications
Film
Broadcast
Surveillance
Object Recognition 3
Key application
Sparse and Dense scene reconstruction
Feature detection &
matching between
pair of views
Multiview
Capture Dense scene
reconstruction
Camera calibration
& Sparse scene
reconstruction
4
Key application
Sparse scene reconstruction
5
(Proposed)
Key application
Dense scene reconstruction
SIFT SFD
6
(Proposed)
Large number of features and matches
Good scene coverage
Improved accuracy
Order of magnitude increase in reconstructed
points.
Why SFD? 7
SFD for dense reconstruction
8
Features Original images Segmented images
Feature matches Sparse reconstruction Dense reconstruction
SFD Algorithm
Over-segmentation:
9
Region boundaries represent lines corresponding to local
maxima of the image function
Watershed segmentation Original image
SFD Algorithm
Feature Detection: Feature Illustration
Feature examples Odzemok segmented image
10
SFD Algorithm
Feature Detection:
11
Evaluation: Datasets
Juggler (6 moving) Odzemok
(6 static, 2 moving)
Cathedral (8 static)
12
Evaluation: Datasets
Valbonne
Merton
Rossendale
13
Evaluation: Features and Matches
Outdoor- Dynamic
14
Matches SFD with Watershed SIFT MSER
Outdoor - Static
15
Matches SFD with Watershed SIFT MSER
Evaluation: Features and Matches
SIFT SFD
Evaluation: Sparse reconstruction
Original
16
261 4084
7211 409
1884 12385
17
Some more results
18
Evaluation: Over-segmentation
SFD: Independent of segmentation technique
Detected features Matches
Evaluation: Over-segmentation
0
2000
4000
6000
8000
10000
12000
Watershed Mean-shift SLIC Watershed Mean-shift SLIC
Merton
Odzemok
19
Detected features Matches
Evaluation: Over-segmentation
0
1000
2000
3000
4000
5000
6000
Watershed Mean-shift SLIC Watershed Mean-shift SLIC
Valbonne
Juggler
20
Evaluation: Matches
0
1000
2000
3000
4000
5000
6000
FAST HARRIS MSER ORB SIFT SURF SFD-WA SFD-MS SFD-SLIC
Odzemok
Merton
Nu
mb
er
of
co
rrect
matc
hes
21
Evaluation: Matches N
um
ber
of
co
rrect
matc
hes
0
500
1000
1500
2000
2500
3000
FAST HARRIS MSER ORB SIFT SURF SFD-WA SFD-MS SFD-SLIC
Valbonne
Juggler
22
Evaluation: Time performance
0
10
20
30
40
50
60
70
FAST HARRIS MSER ORB SIFT SURF SFD-WA SFD-MS SFD-SLIC
Odzemok
Merton
Tim
e p
erf
orm
an
ce i
n m
s
23
Evaluation: Time performance T
ime p
erf
orm
an
ce i
n m
s
0
10
20
30
40
50
60
70
80
90
100
FAST HARRIS MSER ORB SIFT SURF SFD-WA SFD-MS SFD-SLIC
Valbonne
Juggler
24
Mean re-projection error (MRE)
Mean re-projection error (MRE) is calculated for Odzemok dataset for various detectors
N is the number of feature matches
(x,y)
(x`, y`)
3D Ground Truth
Image 1 Image 2
25
Accuracy (MRE) Evaluation of SFD
Feature
Detector
Descriptor RC MRE
SFD SIFT 3717 1.351
SIFT SIFT 1269 1.175
MSER SIFT 119 1.390
FAST BRIEF 121 1.483
Re-projection error of SIFT and SFD-WA for Odzemok
26
Repeatability with camera 1 to all other views (15-120 degree baseline).
Evaluation: Repeatability
Conclusions
Novel feature detector for wide-baseline matching
A comprehensive performance evaluation for feature
matching and time performance
Ground truth accuracy evaluation
Further plans include evaluating the utility of SFD
features in applications such as camera tracking and
object recognition.
28
29
Thank you!
Questions??
Accuracy Evaluation of SFD with Harris
and Uniform Sampling
FD Descriptor Features RC
SFD SIFT 13881 3717
Uniform
Sampling SIFT 12284 33
Harris SIFT 13158 145
• Uniform grid sampling is performed by locating features at points of maximum gradient
magnitude with a 13X13 grid
• Experimented on Odzemok dataset
30
Evaluation: Repeatability
Repeatability between adjacent views (15-30 degree baseline)