1

Unsupervised Modeling and Recognition of Object Categories with Combination of Visual

Contents and Geometric Similarity Links

Unsupervised Modeling and Recognition of Object Categories with Combination of Visual

Contents and Geometric Similarity Links

Gunhee KimChristos Faloutsos

Martial Hebert

Computer ScienceCarnegie Mellon University

October 31, 2008, Vancouver, CanadaACM MIR 2008

2

OutlineOutline

• Problem Statement & Our Approach

• Word Histogram & Network Construction

• pLSA and LDA based Models

• Unsupervised Modeling & Recognition

• Experiments

• Discussion

3

Unsupervised Modeling Unsupervised Modeling

• Category discovery + Ranking

4

Recognition Recognition

Novel Images

Bicycle

Sheep Sign

• Classification + Localization

5

IntuitionIntuition

• Combination of Topic contents and Link AnalysisLatent Topic: Bicycles

Word distributions

Same latent TopicDifferent latent Topic

(Sparse and irregular links)

link distributions

(Dense and consistent links)

link distributions

[1] Sivic, ICCV 2005[2] Fei Fei, ICCV 2005

6

Intuition Intuition

• Combination of Topic contents and link analysis

• Samples of visual words based on Bag-of-Words

• Samples of links generated by image matching

• Two types of evidence into a single generative model

– Ex. Hierarchical Bayesian Models (pLSA, LDA)

7

Our Previous WorkOur Previous Work

• Unsupervised Modeling using Link Analysis Techniques [Kim, CVPR08]

Large Scale Network

Link Analysis Techniques

(ex. PageRank)

- Only links- Only modeling

→ Visual content + Links→ Modeling + Recognition

8

Pros over Conventional Models (1/2)Pros over Conventional Models (1/2)

• Easy Plug-in of geometric information

Indirect Formulation: Link generation with geometric consistency+ Independent of number of parts[Liu, ICCV 2008]

[Lazebnik, CVPR 2006] [Sudderth, ICCV 2005][Niebles, CVPR 2007]

[FeiFei CVPR07 Tutorials]

9

Pros over Conventional Models (2/2)Pros over Conventional Models (2/2)

• Ambiguity in definition of visual words

Word A

Word B

Word C

Semantically similar Different

Different

+ Relaxed by similarity links between words

10

OutlineOutline

• Problem Statement & Our Approach

• Word Histogram & Network Construction

• pLSA and LDA based Models

• Unsupervised Modeling & Recognition

• Experiments

• Discussion

11

Visual Words HistogramVisual Words Histogram

• Follow Standard Bag-of-Words Approach

– Harris Affine + SIFT

– Dictionary Formation: K-mean clustering

Word ID

Freq

Word ID

Freq

Word ID

Freq

: Freq. of word w in the image j (Weighted by Links)

12

Network GenerationNetwork Generation

• Pairwise Image Matching

– Spectral Matching [Leordeanu, ICCV 2005]

: Sum of weights of links from image a to image b

13

OutlineOutline

• Problem Statement & Our Approach

• Word Histogram & Network Construction

• pLSA and LDA based Models

• Unsupervised Modeling & Recognition

• Experiments

• Discussion

14

pLSA Based ModelpLSA Based Model

• Standard pLSA [Hofmann NIPS 1999]

• [Cohn NIPS 2001]

15

LDA Based ModelLDA Based Model

• Standard LDA [Blei JMLR 2003]

wN

z

M

c

Lz

• Linked LDA [Erosheva PNAS 2004]

16

OutlineOutline

• Problem Statement & Our Approach

• Word Histogram & Network Construction

• pLSA and LDA based Models

• Unsupervised Modeling & Recognition

• Experiments

• Discussion

17

Unsupervised Modeling (1/2)Unsupervised Modeling (1/2)

• 1. Category Discovery

– Find out class memberships of all training images

– pLSA based model

– LDA based model

18

Unsupervised Modeling (2/2)Unsupervised Modeling (2/2)

• 2. Ranking

– 1. For recognition, only fixed number of example images to be matched.

O(m) → O(K)

– 2. Highly probable mis-clustering in low ranked images

– pLSA based Model

– LDA based Model

: Most cited documents in topic i

Most matched image in class i

19

New image

RecognitionRecognition

Word ID

Freq

30*K high ranked images

20

RecognitionRecognition

• 3. Classification

– Similar formula to unsupervised clustering.

– pLSA based Model

– LDA based Model

• 4. Localization

– pLSA based Model

– LDA based Model [Sivic ICCV 2005]

21

OutlineOutline

• Problem Statement & Our Approach

• Word Histogram & Network Construction

• pLSA and LDA based Models

• Unsupervised Modeling & Recognition

• Experiments

• Discussion

22

Lea

rne

d ca

tego

ry a

ccu

racy

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

Standard Weighted Linked + Weighted

Comparison TestsComparison Tests

• 5 Objects in Caltech-101

– Similar experimental setup to [Kim CVPR 2008]

55.8% 62.1%

97.8%

23

Unsupervised Category DiscoveryUnsupervised Category Discovery

• Experiment Setup

– MSRC: 6 Objects (75 training / testing)

– PASCAL/ETHZ: 4 objects

(40 training / testing)

85.4 %

90.3 %

PG PM

PC

MB MC

MD MS

MG

PP

24

Classification of Unseen ImagesClassification of Unseen Images

• Experiment Setup

– MSRC: 6 Objects (75 training / testing)

– PASCAL/ETHZ: 4 objects (40 training /

testing)

85.4 %

90.3 %

PG PM

PC

MB MC

MD MS

MG

PP

80.5 %

82.16 %

25

RankingRanking

1.0

0.8

0.6

0.4

0.2

Lea

rne

d ca

tego

ry a

ccu

racy

Number of selected examples per object5 10 15 20 25 30

77.5 % < 85.4 %

Only Link < Link+

Content

26

LocalizationLocalization

• PASCAL/ETHZ dataset

• MSRC dataset

Motorbike Car Peoples Giraffe

Bike Car Sheep Door Sign

27

OutlineOutline

• Problem Statement & Our Approach

• Word Histogram & Network Construction

• pLSA and LDA based Models

• Unsupervised Modeling & Recognition

• Experiments

• Discussion

28

ConclusionConclusion

• Combination of Topic contents and Link Analysis

– Easy Plug-in of geometric information

– Relaxation of the ambiguous definition of visual words

– Integration between two object recognition approaches

• Unsupervised Modeling

+ Recognition

• Competitive performance

29

Comments?Comments?

Thank You

gunhee@cs.cmu.edu