Post on 30-Jun-2015
description
Bundling interest points for object classification
Jordi Sánchez Escué
Supervised byXavier Giró i Nieto
Carles Ventura Royo
Contents● Introduction● State of the art● System Architecture
○ Feature extraction○ Classification○ Evaluation
● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling
● Conclusions & Future work1
Contents● Introduction● State of the art● System Architecture
○ Feature extraction○ Classification○ Evaluation
● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling
● Conclusions & Future work2
Introduction● Does this image contain a plane?
3
Introduction● Does this image contain a plane?
● Which type of flower is it?
4
Introduction● Mobile Visual Search
○ Generalist: Google Goggles
○ Leaf-based: Leafsnap
● Fine-grained classification○ Mushrooms
○ Flowers
5
Introduction● Textures around some interest points
6
Introduction● Features based on regions
7
Introduction● Explore combination: points & regions
8
Introduction● Project Requirements and Goals
○ Comparative study bundling interest points
9
Introduction● Project Requirements and Goals
○ Software Development
10
Contents● Introduction● State of the art● System Architecture
○ Feature extraction○ Classification○ Evaluation
● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling
● Conclusions & Future work11
State of the art● In Defense of Nearest-Neighbor Based
Image Classification, Oren Boiman
12
State of the art● Building contextual visual vocabulary for
large-scale image applications, S. Zhang
13
Contents● Introduction● State of the art● System Architecture
○ Feature extraction○ Classification○ Evaluation
● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling
● Conclusions & Future work14
System Architecture● Interest points and feature extraction
○ Sparse extraction
15
System Architecture● Interest points and feature extraction
○ Interest Points: SURF
16
System Architecture● Binary Partition Tree (BPT)
○ Partition: 20 reg. SLIC
17
● Binary Partition Tree○ A scale is chosen (ex, N = 3)
System Architecture
18
Contents● Introduction● State of the art● System Architecture
○ Feature extraction○ Classification○ Evaluation
● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling
● Conclusions & Future work19
System Architecture● Classification: Training
20
Trainer
1
2
3
4
System Architecture
CLASSIFIER1-NN, euclidean
distance
1
3
4
2
● Classification: Detection
21
Target image
System Architecture
Query image
22
System Architecture
Target image
Query image
23
System Architecture
Query image
Target image
24
System Architecture
Query image
Target image
25
System Architecture
Query image
Target image
26
System Architecture
Query image
Target image
27
System Architecture
Query image
Nearest Target image11
28
Contents● Introduction● State of the art● System Architecture
○ Feature extraction○ Classification○ Evaluation
● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling
● Conclusions & Future work29
System Architecture● Evaluation
○ Development of an evaluation tool
30
Tools
System Architecture● Software development
31
Trainer
Detector
Evaluation
SVM adapted to a flexible architecture
New tool for evaluation
Can be adapted to any classifier
or descriptor
Contents● Introduction● State of the art● System Architecture
○ Feature extraction○ Classification○ Evaluation
● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling
● Conclusions & Future work32
M-E. Nilsback & A. Zisserman, «A Visual Vocabulary for Flower Classification» Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2006. http://www.robots.ox.ac.uk/~vgg/data/flowers/17/
33
0.591769
0.3813720.463660
Experiments: basic approach● Results
34
Contents● Introduction● State of the art● System Architecture
○ Feature extraction○ Classification○ Evaluation
● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling
● Conclusions & Future work35
Experiments: Class aggregation● Aggregation of the interest points of all the
images of the same class to do the matching
36
Experiments: Class aggregation● Results
37
0.59
0.380.46
0.78
0.43
0.56
Contents● Introduction● State of the art● System Architecture
○ Feature extraction○ Classification○ Evaluation
● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling
● Conclusions & Future work38
● Region restriction
Experiments: Bundling interest points
39
Experiments: Bundling interest points
40
● Why the results did not improve?○ Image flower segmentation
Experiments: Bundling interest points
41
● Why the results did not improve?○ Bad flower segmentation (N = 2)
Experiments: Bundling interest points
42
● Why the results did not improve?○ Bad flower segmentation (N = 2)
● Future work to improve results○ Using perfect manual segmentation
Experiments: Bundling interest points
43
● Why the results did not improve?○ Good region matching (flower to flower)
Experiments: Bundling interest points
44
● Why the results did not improve?○ Bad region matching (flower to background)
Experiments: Bundling interest points
45
● Why the results did not improve?○ Bad region matching (flower to background)
● Future work to improve results○ Avoid using edge regions
○ Using object candidates
Experiments: Bundling interest points
46
Contents● Introduction● State of the art● System Architecture
○ Feature extraction○ Classification○ Evaluation
● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling
● Conclusions & Future work47
Experiments: Class aggregation & Bundling
● Class aggregation with points bundled in regions
48
● Comparative study
Experiments: Class aggregation & Bundling
49
Contents● Introduction● State of the art● System Architecture
○ Feature extraction○ Classification○ Evaluation
● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling
● Conclusions & Future work50
Conclusions & Future Work● Comparative study done
○ Bundling interest points into regions worsens the F1-score between 1% and 7%
○ Class aggregation improves the F1-score by 9.2%
● State of the art comparative study
○ Pointless having bad results
● Software development
● Future Work51
Bundling interest points for object classification
Jordi Sánchez Escué
Supervised byXavier Giró i Nieto
Carles Ventura Royo
System Architecture● Classification: Training
○ Semantic annotation & Ontology
......
...
...
...
...
1
2
3
4
System Architecture● Binary Partition Tree (BPT)
○ 20 SLIC superpixels
Future work● Add new approaches
○ Class aggregation in the query
○ Bundling query image, not bundling target
images (with certain spatial restriction).
● Optimize k, change classifier, more descriptors