Post on 01-Jan-2016
ACTIVE LEARNING USING CONFORMAL PREDICTORS: APPLICATION TO IMAGE
CLASSIFICATION
Hyp Introduction Hyp Conceptual overview
Hyp Experiments and resultsHyp Conclusions
L. Makili1, J. Vega2, S. Dormido-Canto3
1Universidade Katyavala Bwila. Benguela (Angola)2Asociación EURATOM/CIEMAT para Fusión. Madrid (Spain)
3Universidad Nacional de Educación a Distancia (UNED). Madrid (Spain)
7th Workshop on Fusion Data Processing Validation and Analysis
Outline
• Introduction• Concepts overview• Experimental results• Conclusions
Hyp IntroductionHyp Conceptual overview
Hyp Experiments and resultsHyp Conclusions
Motivation • 5 – class classification problem
– Classification of TJ – II Thomson Scattering images
• Classifier based on conformal predictors, using SVM as the underlying algorithm– Computationally intensive task
Patterns of TSD images: (a) BKGND, (b) COFF, (c) ECRH, (d) NBI and (e) STRAY
Hyp IntroductionHyp Conceptual overview
Hyp Experiments and resultsHyp Conclusions
Goal
• To find out a minimal and good enough training dataset for classification purposes
Hyp IntroductionHyp Conceptual overview
Hyp Experiments and resultsHyp Conclusions
Active learning• The learning algorithm must have some
control over the data from which it learns• It must be able to query an oracle, requesting
for labels of data samples that seem to be most informative for the learning process
• Proper selection of samples implies better performances with fewer data
Hyp IntroductionHyp Conceptual overview
Hyp Experiments and resultsHyp Conclusions
Settles, B. “Active Learning Literature Survey. Computer Sciences Technical Report 1648”, University of Wisconsin – Madison, 2009. Available at http://research.cs.wisc.edu/tech reports/2009/TR1648.pdf
Uncertainty sampling
• The learning algorithm selects new examples when their class membership is unclear
• Suitable for classifiers that besides making classification decisions, estimates certainty of these decisions
Hyp IntroductionHyp Conceptual overview
Hyp Experiments and resultsHyp Conclusions
Lewis, D. and Gale, W., “A Sequential Algorithm for Training Text Classifiers”. In Proceedings of the ACM – SIGIR Conference on Research and Development in Information Retrieval, Croft, W. B. and van Rijbergen, C. J. (eds). New York: Springer – Verlag, 1994, pp. 3 – 12
Conformal prediction• Permits complementation of predictions made
by machine learning algorithms with some measures of reliability
• Besides the label predicted for a new object, it outputs two additional values– Confidence– Credibility
Hyp IntroductionHyp Conceptual overview
Hyp Experiments and resultsHyp Conclusions
Vovk, V., Gammerman, A. and Shafer, G., Algorithmic Learning in a Random World, New York: Springer Science + Business Media, Inc., 2005
Conformal prediction
• Used as nonconformity scores the Lagrange multipliers computed during SVM training
• Extended to a multiclass framework in a one-vs-rest approach
Hyp IntroductionHyp Conceptual overview
Hyp Experiments and resultsHyp Conclusions
Active learning algorithm• Inputs
– Initial training set T, calibration set C, pool of candidate samples U
– Selection treshold τ, batch size β• Train an initial classifier on T• While a stopping-criterion is not reached
– Apply the current classifier to the pool of samples– Rank the samples in the pool using the uncertainty criterion– Select the top β examples whose certainty level fall under the
selection threshold τ– Ask teacher to label the selected samples and add them to the
training set– Train a new classifier on the expanded training set
Hyp IntroductionHyp Conceptual overview
Hyp Experiments and resultsHyp Conclusions
(Un)Certainty criteria• Credibility:
• Confidence:
• Query-by-transduction:
• Multicriteria:
– Combination 1:
– Combination 2:
credIcr )(x
confIcf )(x
)1()( confcredIqbt x
)()1()()( 21 xxx III
78.0);()1()()( xxx cfqbt III
5.0);()1()()( xxx cfqbt III
Hyp IntroductionHyp Conceptual overview
Hyp Experiments and resultsHyp Conclusions
Ho, S. and Wechsler, H., “Query by Transduction”. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30 (9), 2008, pp. 1557 – 1571
Setup • 1149 samples divided into
– Initial training: 5 samples– Pool: 794 samples– Calibration set: 150 samples– Test set: 200 samples
• Batch size: 25 • Selection treshold : 0.4
• Used with Qbt, Comb1 and Comb2
Hyp IntroductionHyp Conceptual overview
Hyp Experiments and resultsHyp Conclusions
Setup • For Each criterion in (Qbt, Comb1, Comb2)
– For experiment = 1 : 10• Select test set• Run active learning algorithm selecting 700 samples hybridly• For NumTrn = 50 : 50 : 700
– Train CP classifier on the first NumTrn samples – Aplly classifier to the test set
• End for NumTrn
– End For experiment• End For Each
Hyp IntroductionHyp Conceptual overview
Hyp Experiments and resultsHyp Conclusions
Conclusions• Active learning was applied to the selection of a
minimal and good enough training dataset for classification purposes
• It allows reaching higher success rates and confidence in predictions with fewer data points, compared to the random selection of the training set
• Combining multiple criteria we can balance the trade-off between success rate and confidence of prediction improvement
Hyp IntroductionHyp Conceptual overview
Hyp Experiments and resultsHyp Conclusions