Towards Open World Recognition

Abhijit Bendale, Terrance BoultUniversity of Colorado of Colorado Springs

Poster no 85

2

Multi-Class Classification System

W Scheirer, A Rocha, A Sapkota, T Boult “Towards Open Set Recognition” IEEE TPAMI 2013

3

Out in the Real-WorldDetect New Category

Pool Table

Bowling Pin

Boxing glove

Calculator

Chess board

?

?

?

?

4

Open World Recognition

• World with Knowns (K) &Unknowns Unknowns (UU)

Detect as Unknown

• NU: Novel Unknowns

Label Data• LU: Labeled

Unknowns

Incremental Class Learning

• K: Known

Scale

5

Open Set Learning

Incremental Learning

Scalable Learning

Ristin+ (CVPR’14)

Yeh+ (CVPR’08)

Li+ CVPR’07)

Mensink+(PAMI’13)

Related Work

Jain+ (ECCV’14)

Scheirer+ PAMI’13)

Scheirer+(PAMI’14)

Deng+ (NIPS’11)

Marszalek+(ECCV’08)

Liu+ (CVPR’13)

Open World Recognition

6

Open Space in Classification

??

?

?

?

?

? ? ??

??

??

Open Set Recognition Empirical risk

functionRegularization constant

Open space risk

W Scheirer, A Rocha, A Sapkota, T Boult “Towards Open Set Recognition” IEEE TPAMI 2013

Closed SpaceOpen Space

7

NCM – Metric Learning

NCM Classifier with Metric Learning

T Mensink, J Verbeek, F Perronin, G Csurka “Distance based Image Classification: Generalizing to New Classes at Near Zero Cost” IEEE TPAMI 2013M Ristin, M Guillaumin, J Gall, L Van Gool “Incremental Learning of NCM Forests for Large-Scale Image Classification” CVPR 2014

8

Compact Abating Probability (CAP) Models

W Scheirer, L Jain, T Boult “Probability Models for Open Set Recognition” IEEE TPAMI 2014

Class Mean

9

Theorem 1: Open Space Risk for Model Combination

10

Theorem 2: Open Space Risk for Transformed Spaces

11

Opening an Existing Algorithm:Nearest Non-Outlier (NNO) Algorithm

W = Linear Transformation (weight matrix from metric learning)

Standard gamma functionIn volume of m-D ball Class mean for class i

τ is threshold for open world

Probability

12

Open World Evaluation

Training phase

Testing phase

Parameter Learning Phase Incremental Learning Phase

Closed Set TestingUnknown Categories

Known Categories

Open Set Testing

? ?

13

Training for Open World

NCM - ML NNO

• Parameter Learning with initial set of categories

• Estimation of τ for open set learning to balance open space risk• Optimize for Known vs Unknown Errors• Incrementally add new categories

14

ExperimentsDatasets

• ILSVRC’10: 1.2M training images, 1000 classes• ILSVRC’12: 1.2M training images, 1000 classes

Features• Dense SIFT features, Quantized into 1000 Bag of Visual Words• Publically available features• LBP, HOG, Dense SIFT (for ILSVRC’12)

Algorithms• Nearest Class Mean - ML Classifier (NCM) [Mensink etal PAMI 2013]• Nearest Non-Outlier Algorithm (NNO) [This Paper]• 1vSet [Scheirer etal PAMI 2013]• Linear SVM [Liblinear, Fan etal JMLR 2008]

50 Initial Categories

Increasing # of unknown categories during testing i.e. increasing openness of problem

Incrementally addingcategories during training

Closed Settesting

16

200 Initial Categories

Increasing # of unknown categories during testing i.e. increasing openness of problem

Incrementally addingcategories during training

Closed Settesting

500 known + 500 unknown categories

17

• Formalized Open World Recognition and showed how to “Open” an existing algorithm.

• NNO allows construction of scalable systems that can be updated incrementally

Conclusion & Future Work

See us at Poster no 85 …!!!!

• Exploring sophisticated novelty detectors, open world detection, “opening” other baseline algorithms etc.

• Open World Deep Learning methods• Happy to Collaborate…!!!