Hierarchical Subquery Evaluation for Active Learning on a Graph
80
Hierarchical Subquery Evaluation for Active Learning on a Graph Oisin Mac Aodha, Neill Campbell, Jan Kautz, Gabriel Brostow CVPR 2014 University College London 1
description
Hierarchical Subquery Evaluation for Active Learning on a Graph. CVPR 2014. Oisin Mac Aodha , Neill Campbell, Jan Kautz , Gabriel Brostow. University College London. Large Image Collections. Cat. Dog. Horse. https:// www.flickr.com/photos/cmichel67. Large Image Collections. Cat. - PowerPoint PPT Presentation
Transcript of Hierarchical Subquery Evaluation for Active Learning on a Graph
1
Hierarchical Subquery Evaluation for Active Learning on a Graph
Oisin Mac Aodha, Neill Campbell, Jan Kautz, Gabriel Brostow
CVPR 2014
University College London
2
CatDogHorse
Large Image Collections
https://www.flickr.com/photos/cmichel67
3
Large Image Collections
https://www.flickr.com/photos/cmichel67
CatDogHorse
Labeling large image collections is tedious
4
Acquiring Annotations
https://www.flickr.com/photos/usnavy https://www.flickr.com/photos/rdecom
Crowdsourcing Specialized Knowledge
Expert time is valuable!
5
Active Learning
Oracle
AL Algorithm
User Query
Label
UnlabeledDataset
6
Number of user queries
TestAccuracy
1
0
Learning Curves
7
Number of user queries
1
0
Learning Curves
TestAccuracy
8
Number of user queries
1
0
Learning Curves
TestAccuracy
9
Number of user queries
1
0
Learning Curves
TestAccuracy
10
Learning Curves
Number of user queries
1
0
TestAccuracy
11
Learning Curves
Number of user queries
1
0
We want the largest area under the learning curve
TestAccuracy
12
Learning Curves
1
0
TestAccuracy
The number of unlabeled images can be very large!
13
Active Learning Wish List
14
• Fast updating of classifier for interactive labeling
Active Learning Wish List
15
• Fast updating of classifier for interactive labeling• Exploit structure in unlabeled data
Active Learning Wish List
16
• Fast updating of classifier for interactive labeling• Exploit structure in unlabeled data• Consistent performance across different datasets
Active Learning Wish List
17
• Fast updating of classifier for interactive labeling• Exploit structure in unlabeled data• Consistent performance across different datasets• Make the most of the expert’s time
Active Learning Wish ListGraph Based
Semi-Supervised Learning
Perplexity Graph Construction
Our Hierarchical Subquery Evaluation
18
Related Work
Video SegmentationFathi et al. BMVC 2011
Action DetectionBandla and Grauman ICCV 2013
Gaussian Random FieldsZhu et al. ICML 2003
Semantic SegmentationVezhnevets et al. CVPR 2012
RALF: Reinforced Active LearningEbert et al. CVPR 2012
…
Image ClassificationKapoor et al. ICCV 2007
…
19
xiφ( ) =
Supervised Classification
20
xjφ( ) =
Supervised Classification
21
Supervised Classification
22
Supervised Classification
Decision Boundary
23
Semi-supervised learning using Gaussian fields and harmonic functions X. Zhu, Z. Ghahramani, J. LaffertyICML 2003
Fi = P(f(xi) == class1)
wij
Semi-Supervised Learning
24
Semi-Supervised LearningFi = P(f(xi) == class1)
wij
25
Graph Construction
Stochastic neighbor embeddingG. Hinton and S. RoweisNIPS 2002
26
Graph Active Learning
27
Example 2 Class Graph
28
Example 2 Class Graph
Ground Truth
29
Example 2 Class GraphActive Learning Strategies
30
Active Learning Strategies
• Random
31
Active Learning Strategies
• Random• Exploration – clusters
32
Active Learning Strategies
• Random• Exploration – clusters• Exploitation – uncertainty
33
Active Learning Strategies
• Random• Exploration – clusters• Exploitation – uncertainty
34
Active Learning Strategies
• Random• Exploration – clusters• Exploitation – uncertainty• RALF – explore or exploit
Ralf: A reinforced active learning formulation for object class recognitionS. Ebert, M. Fritz, and B. SchieleCVPR 2012
35
Active Learning Strategies
• Random• Exploration – clusters• Exploitation – uncertainty• RALF – explore or exploit• Expected Error Reduction – reduce future
error
Toward optimal active learning through sampling estimation of error reductionN. Roy and A. McCallum ICML 2001
37
Expected Error Reduction
2 Labeled Points
Ground Truth
38
Expected Error Reduction
Current ClassDistribution
Ground Truth
39
Expected Error Reduction
Compute the Expected Error (EE) for each unlabled datapoint
Ground Truth
40
Expected Error Reduction
? Hypothesize label 1
Ground Truth
Class 1 Class 2
41
Expected Error Reduction
? Update model
Ground Truth
42
Expected Error Reduction
? Hypothesize label 2
Ground Truth
Class 1 Class 2
43
Expected Error Reduction
? Update model
Ground Truth
44
Expected Error Reduction
? Compute EE
Ground Truth
45
Expected Error Reduction
?
Hypothesize label 1
Ground Truth
Class 1 Class 2
46
Expected Error Reduction
?
Update model
Ground Truth
47
Expected Error Reduction
?
Hypothesize label 2
Ground Truth
Class 1 Class 2
48
Expected Error Reduction
?
Update Model
Ground Truth
49
Expected Error Reduction
?
Compute EE
Ground Truth
51
Expected Error Reduction
Repeat for all unlabeled
nodes!O(N2)For Zhu et al.
Ground Truth
52
Problems with EER
• Need to retrain the classifier with each unlabeled example (subquery) and for each different class label – O(N2)
At each step is it necessary to try every possible subquery?
53
Active Learning Strategies
Lower Complexity
Performance RALFCVPR 2012
EERZhu 2003
Random
54
Unsupervised Hierarchical Clustering
Unsupervised Hierarchical Clustering
55
Authority-shift clustering: Hierarchical clustering by authority seeking on graphsM. Cho and K. Mu LeeCVPR 2010
…
Unsupervised Hierarchical Clustering
56
…
Unsupervised Hierarchical Clustering
57
…
Unsupervised Hierarchical Clustering
58
Large clusters (exploration)
Boundary refinement (exploitation) …
Our Hierarchical Subquery Evaluation
After 2 Queries
59
Ground Truth
Our Hierarchical Subquery Evaluation
5.6 4.2
3.5After 2 Queries
Best EE
Next nodes to add to the active set
CurrentActive Set
60
Ground TruthRemaining Subqueries: 74
Our Hierarchical Subquery Evaluation
Best EE
After 2 Queries
61
Ground Truth
6 2.15.6
3.5
4.2
Remaining Subqueries: 2
Our Hierarchical Subquery Evaluation
6 2.1
3.21.1
After 2 Queries
62
Ground Truth
5.6
3.5
4.2
Remaining Subqueries: 0
Our Hierarchical Subquery Evaluation
6 2.1After 3 Queries
3.21.1
Label for the example with the best EE is requested
After 2 Queries
63
Ground Truth
5.6
3.5
4.2
Remaining Subqueries: 0
Our Hierarchical Subquery Evaluation
After 3 Queries
After 2 Queries
64
Ground TruthRemaining Subqueries: 72
65
Results
66
Results
1579 examples8 classes50 dim BoW PCA
67
Results
68
Results
Ralf: A reinforced active learning formulation for object class recognitionS. Ebert, M. Fritz, and B. SchieleCVPR 2012
69
Results
70
13 Different Computer Vision and Machine Learning Datasets
Results - Area Under Learning Curve
71
13 Different Computer Vision and Machine Learning Datasets
Results - Area Under Learning Curve
72
Summary
• Hierarchical graph based semi-supervised active learning O(N2) -> O(NlogN)
73
Summary
• Hierarchical graph based semi-supervised active learning O(N2) -> O(NlogN)
• Robust to dataset type
74
Summary
• Hierarchical graph based semi-supervised active learning O(N2) -> O(NlogN)
• Robust to dataset type • Best user query in the time available
75
Future Work
• Representation learning – update graph structure during labeling
76
• Representation learning – update graph structure during labeling
• Model different annotation costs
Future Work
77
• Representation learning – update graph structure during labeling
• Model different annotation costs• Embed new datapoints into the graph
Future Work
79
Come visit our poster 01-C-3
http://visual.cs.ucl.ac.uk/pubs/graphActiveLearning
80
81
Graph Construction Comparison
82
Timings
83
