Spatially coherent latent topic modelfor concurrent object segmentation and classification

Liangliang CaoDept. of Electrical and Computer Engineering

University of Illinois at Urbana-Champaign (UIUC), USAcao4@ifp.uiuc.edu

Li Fei-FeiDept. of Computer SciencePrinceton University, USAfeifeili@cs.princeton.edu

Abstract

We present a novel generative model for simultaneouslyrecognizing and segmenting object and scene classes. Ourmodel is inspired by the traditional bag of words represen-tation of texts and images as well as a number of relatedgenerative models, including probabilistic Latent SematicAnalysis (pLSA) and Latent Dirichlet Allocation (LDA). Amajor drawback of the pLSA and LDA models is the as-sumption that each patch in the image is independently gen-erated given its corresponding latent topic. While such rep-resentation provide an efficient computational method, itlacks the power to describe the visually coherent imagesand scenes. Instead, we propose a spatially coherent la-tent topic model (Spatial-LTM). Spatial-LTM represents animage containing objects in a hierarchical way by over-segmented image regions of homogeneous appearances andthe salient image patches within the regions. Only one sin-gle latent topic is assigned to the image patches within eachregion, enforcing the spatial coherency of the model. Thisidea gives rise to the following merits of Spatial-LTM: (1)Spatial-LTM provides a unified representation for spatiallycoherent bag of words topic models; (2) Spatial-LTM cansimultaneously segment and classify objects, even in thecase of occlusion and multiple instances; and (3) Spatial-LTM can be trained either unsupervised or supervised, aswell as when partial object labels are provided. We verifythe success of our model in a number of segmentation andclassification experiments.

1. Introduction

Understanding images and their semantic contents is animportant and challenging problem in computer vision. Inthis work, we are especially interested in simultaneouslylearning object/scene category models and performing seg-mentation on the detected objects. We present a novel gen-erative model for both unsupervised and supervised classi-fication.

In recent years, bag of words models (e.g., probabilistic

A. Input Image B. LDA initialized topics C. LDA learned topics

E. Coherent regions for

Spatial-LTM

F. Spatial-LTM

initialized topics

G. Spatial-LTM

learned topics

H. Spatial-LTM

segmentation results

D. LDA learned object

Figure 1. (This figure must be viewed in colors) Comparison ofthe bag of words model and our proposed model, Spatial-LTM.Panel A shows the original input image. Panel E shows the inputregions to Spatial-LTM provided by a over-segmentation method.Panels B and F in Column 2 illustrate the initial latent topicas-signments to the image patches by either the traditional bagofwords model (LDA) or Spatial-LTM. The circles denote the visualwords provided by local interest point detectors. Different colorsof the circles denote different latent topics of the patch. Panels Cand G in Column 3 illustrate the latent topic assignments after themodels have learned the object classes. Most of the patches onthe human face are colored in red, indicating that the modelshavesuccessfully recognized object. Finally, Panel D and H in Column4 show the segmentation results of LDA and Spatial-LTM respec-tively. One can see that by encoding the spatial coherency oftheimage patches, Spatial-LTM achieves a much better segmentationof the object.

Latent Sematic Analysis (pLSA) [13] and Latent Dirich-let Allocation (LDA) [2]) have shown much success fortext analysis and information retrieval. Inspired by this,a number of computer vision works have demonstratedimpressive results for image analysis and classification[6, 7, 27, 11, 25, 22, 20] using the bag of words models.Such models are attractive due to the computational effi-ciency and their ability of representing images and objectswith dense patches. This is achieved by an assumption inbag of words models where the spatial relationships of theimage patches or object parts are ignored. In contrast, object

models that explicitly encode spatial information among theparts or regions usually require formidable computationalcosts in inference [12, 10, 16], forcing these algorithms tooften represent objects with a very few number of parts.

One should observe, however, there is a distinct differ-ence between image and text analysis: there are no natu-ral visual analogues of words. In other words, text doc-uments are naturally composed of distinctive vocabularieswhile there is no such obvious word-level representation inimages. To borrow the algorithms from text analysis, theresearchers usually employ various local detectors [15, 23],and describe them as “visual words” that play the role of vo-cabulary words in the bag of words model. Although mod-ern local detectors and descriptors offer powerful character-ization of images, the bag of visual words representation re-sulted from these detectors have some inherent weaknesses.First, detected regions on an image are often sparse, leavingmany uncovered areas of the images. When one is inter-ested in segmentation, this kind of sparsity is detrimentalto the eventual results. Second, spatial relationships amongthe different parts of an image are often critical in visualrecognition tasks. A scrambled collection of patches froma car image does not necessarily evoke the recognition ofa car. The current bag of words representation ignores thisimportant issue, hence affecting the final accuracy of therecognition tasks.

In this paper, we would like to inherit the strength ofthe bag of words representation, and improve it by incor-porating meaningful spatial coherency among the patches.In traditional bag of words models, one word is assigned alatent topic independently. This often results in segmenta-tion results similar to that of Panel G in Fig.1. In contrast,our model believes thatpixels should share the the samelatent topic assignment if they are in a neighboring regionwith similar appearance. In other words, the latent topicsassignments of the pixels in an image are spatially coherentin our model, whereas they are independent of each other inthe traditional bag of words model. Formally, we call ourmodel thespatially coherent latent topic model, or Spatial-LTM for short.

A number of approaches recently have looked at theissue of simultaneous classification and segmentation. Acomparison of our work and the related works is given inTable1. In this work, we would like to design a probabilisticframework toward general object segmentation and classifi-cation. Our model provides a way of unifying color/texturefeatures with visual words at a low computational cost. Fur-thermore, Spatial-LTM is tolerant to shape deformation andtransformation without sacrificing the computational effi-ciency to model the spatial freedom. In this way, we differfrom the shape based segmentation works in [4, 31, 16] andclassification works in [1, 18, 17, 11, 5]. Our work is alsorelated to Russell et. al’s recent work on object segmen-

tation [25]. Both works employ visual words for the taskof image segmentation. However, our Spatial-LTM modelis fundamentally a nested new representation of the imagesand objects as oppose to the pLSA/LDA model in [25]. Ourmodel generates the topic distribution at the region levelinstead of the word level as in [25]. Moreover, Our worktreats each image as one document, while [25] treats eachimage segment as one separate document. This fundamen-tal difference enables us to recognize and segment occludedobjects, whereas Russell et al. [25] cannot do so.

In summary, the contributions of Spatial-LTM are: (1)Spatial-LTM provides a unified representation for spatiallycoherent bag of words topic models; (2) Spatial-LTM cansimultaneously segment and classify objects, even in thecase of occlusion and multiple instances; and (3) Spatial-LTM can be trained either unsupervised or supervised, aswell as when partial object labels are provided. Fig.1 illus-trates Spatial-LTM’s novelty by comparing the segmenta-tion and recognition results of an image by both the LDA(one popular bag of words model) and Spatial-LTM.

The rest of the paper is organized as follows: Section2introduces a nested representation of the images, which isa region-based data structure for our model. In Section3,we develop Spatial-LTM, a novel generative model of la-tent topics and visual words and discuss the inference andtraining of this model. Note that our model is fit for bothunsupervised learning and supervised learning. Section4shows the experimental results of Spatial-LTM under thesetwo settings.

2. Image Representation for Spatial-LTM

Given an unknown and unlabeled image, Spatial-LTMaims to simultaneously recognize categories of objects inthe scene as well as segment out the objects. To explicitlymodel the spatially coherent nature of images, we enforcethe pixels to share the same latent topic within a homoge-neous region. Here homogeneous region means that the pix-els in the region are similar with respect to some appearancefeature, such as intensity, color, or texture. The visual wordsin the region along with the region’s overall appearance arein turn discriminant for object recognition. In Section3 wewill show the detailed generative model that represents thishierarchical relationship between patches and regions.

As a first step of our algorithm, one starts with an ini-tial over-segmentation of an image by partitioning an im-age into multiple homogeneous regions. Here we chooseto use a segmentation algorithm proposed by Felzenszwalband Huttenlocher [9], which incrementally merges regionsof similar appearance with small minimum spanning treeweight. This method is of nearly linear computational com-plexity in the number of neighboring pixels. In this paperwe use a modified version of the algorithm in [9] to obtaincoherent regions of homogeneous appearance. It is worth

worksegmentation

classification

need for labeling

occlusion

multi-object

Spatial-LTM√√√ √√√ ∼∼∼ ∗

√√√ √√√

Sivic et al. [27] × √ ∼ √ √

Rotheret al. [24]√ × ∼ √ √

Todorovicet al. [28]√ √ ∼ √ √

Russellet al. [25]√ √ ∼ √ √

LOCUS [31]√ √ ∼ × ×

Leibeet al. [18, 17]√ √

image label√ √

ObjCut [16]√ × image label

√ ×Borensteinet al. [4, 3]

√ √pixel label × ×

CRF [19]√ √

pixel label√ √

Table 1. Comparison of other works that combine classification and segmentation for object recognition.∗Our model is fit for both unsupervised andsemi-supervised learning.

mentioning that our model is not tied to a specific segmen-tation algorithm. Any method that could propose a reason-able over-segmentation of the images would suit our needs.

We choose colors (inLab space) or texture features [26]to describe the appearance of a region. To avoid obtainingregions larger than the objects we want to classify, we startwith an over-segmentation of the images. This means welet the algorithm partition an image into roughly30 ∼ 50homogeneous regions. When the segments number by theoriginal algorithm in [9] is too small (less than 30) on animage, we constraint the algorithm by forbidding it to mergeregions larger than a threshold (we choose 3600 as thresholdfor 400 × 300 images). On a computer with Intel 2.16GCPU, the algorithm takes less than0.5 second to segmentone image of the size400 × 300 pixels. Fig.2 shows oneexample of our modified approach.

It is worth noting that our segmentation step serves afundamentally different role compared to the segmentationin [25]. In [25], segments of the images are treated as sepa-rate documents. Each segment is assumed to be a potentialobject in its entirety. In our case, we use the over-segmentedpieces as building blocks for the hierarchical model. Ourrepresentation allows further integration of these segmentssuch that our algorithm could recognize objects that are

Figure 2. Example image to illustrate regions of homogeneousappearance. Left: original image. Center: segmentation bytheoriginal algorithm [9]. Right: segmentation by our modified al-gorithm. Over-segmentation of an image is desirable for Spatial-LTM because regions can easily be merged into the same topicduring learning.

heavily occluded (see Section4). In addition, this way ofusing the image segments makes our algorithm less vulnera-ble to the quality of the segmentation step compared to [25].

After the over-segmentation step, we represent an im-age in a region-based structure for the Spatial-LTM model.The appearance feature of each region is characterized bythe average of color or texture features over all the pixelswithin the region. Within each region, we find a number ofinterest points by using the scale invariant saliency detec-tor [15]. Each interest point is described by SIFT [23]. Twocodebooks are obtained for both the interest point patches(or visual words) and the region appearance, with the sizeW andA, respectively. This is done by using unsupervisedk-means clustering.

Fig. 3 shows the data structure that represents an image.For an imageId, we obtainRd regions. Each regionr hasone homogeneous appearance featurear, and a set of ofvisual wordswi

r, where1 ≤ i ≤ Mr. Notear andwir take

discrete values of{1, 2, ..., A} and{1, 2, ..., W} accordingto the respective codebook. We denote all the words withinthe regionr as a vectorwr = {w1

r , w2r , ..., w

Mr

r }.

Region Region Region

Image Id r=1 r=2 r=Rd...

ar wr

appearance visual words

...wr1

wr

Mrwr

i

wr2

...

Figure 3. The region-based image representation for spatial-LTM.The image is partitioned intoRd regions, each regionr has onehomogeneous appearance featurear, and a set of of visual wordswi

r, where1 ≤ i ≤ Mr.

3. The Generative Model of Spatial-LTM

In this section, we introduce the Spatial-LTM model andcompare its difference with the traditional LDA model. Wewill also discuss how to learn the parameters for Spatial-LTM in section3.1. To make the presentation clearer, wefirst consider Spatial-LTM in the unsupervised scenario andthen generalize it to the supervised version in section3.2.

Given an imageId and its partitioned regionsr =1, 2, ..., Rd, we uselatent topiczr to represent the labelsof regionr. Such label represents category identities of dif-ferent objects, e.g., animals, grass, cars, background, andetc. Suppose there areK topics within the image collection,then for each region,zr = 1, 2, ..., K. For the segmentationtask, we need to infer latent topiczr for each region andgroup all the regions with the samezr into one object. Forthe classification task, we choose the latent topic with thehighest probability as the category label of the image.

Our generative model behaves as following: first wedraw a distribution of topics for each imageId, representedby θθθd. As in [2], the prior distribution ofθθθd is described bya Dirichlet distribution with parameterλ. Givenθθθd, we se-lect a topiczr for each region in the image. Given regionr

and its correspondingzr, we choose the overall appearanceof the region (either color or texture) according to a distri-bution governed byααα. Finally, for each of theMr patcheswithin the regionr, we draw a visual codeword to representthe patch according to the topic distributionβββ. Fig.4(a) and(b) compares the different graphical model representationof LDA and Spatial-LTM for unsupervised learning.

The joint distribution of{ar,wr, zr} given an imageId

can be written as

Pr(ar,wr, zr|λ, α, β) (1)

= Pr(θd|λ)Pr(zr |θ)Pr(ar |zr, α)

Mr∏

i=1

Pr(wir |zr, β)

whereα, β are parameters describing the probability of gen-erating appearance and visual words given the topic, respec-tively. Pr(zr |θ) is a Multinomial distribution andPr(θ|λ)is aK-dimensional Dirichlet distribution.

Then the likelihood of a single image for Spatial-LTM is

Ld = log

Z

dθd

RdY

r=1

Dir(θd|λ)Pr(ar,wr|α, β, θd) (2)

= log

Z

dθd

RdY

r=1

X

zr

Dir(θd|λ)Pr(ar,wr, zr|θd, α, β)

In the training step, we maximize the total likelihood ofthe training imagesL =

∑D

d=1 Ld subject to all the modelvariablesλ, α, θ andzr. In the testing step, the learnedλ, α

are fixed. We need only to estimate the parameterθd and

zr’s for each image. For the classification task, a new imageis classified as objectk∗ if

k∗ = arg max1≤k≤K

θd(k) (3)

For the segmentation task, we label the regionr with z∗rsuch that

z∗r = argmaxzr

Pr(ar,wr|zr) (4)

The regions with the specificz∗r constitute the interested ob-ject.

In practice it is intractable to maximizeL directly due tothe coupling between hidden variablesθ, α, β. To estimatethe parameters and hidden states, we employ variational in-ference methods [14, 29] to maximize the lowerbound ofL.

To make the expression more compact, we employ thenotation as in [29]. We denote the visible nodes in Fig.4(w,a) asV and the invisible nodes (z, θ, α, β) asH. Toapproximate the true posterior distributionP (H|V), we in-troduce the variational distributionQ(H). We obtain thevariational lowerbound for the likelihood

lnL =X

H

Q(H) lnPr(H,V)

Q(H)+ KL(Q|P )

≥X

H

Q(H) lnPr(H,V)

Q(H)= LQ (5)

whereKL(Q|P ) ≥ 0 is the Kullback-Leibler divergence.

(b)

c

a

w

DRd

WxK

β

AxK

α

Mr

z

(a)

θλ

λC

a

w

DRd

WxK

β

AxK

α

Mr

zθ

(c)

w

D

WxK

βzθλMd

Figure 4. Graphical model representation of Spatial-LTM and thecomparison with a traditional bag of words mode LDA. (a) La-tent Dirichlet Allocation model (LDA). (b) Unsupervised Spatial-LTM. (c) Supervised Spatial-LTM. The cyan and the red framesdenote groups of visual words and homogeneous regions, respec-tively. The shaded circles stand for the observations from the im-age, while the others are variables to be inferred.

Maximizing L is intractable, we therefore choose tomaximizeLQ to approximately estimate the parameters.In Section3.1 we introduce the framework of Winn andBishop’s Variational Message Passing (VMP) for our varia-tional estimation [29].

3.1. Model Inference using VMP

In this section we discuss how to maximize the varia-tional lower boundLQ. Suppose we could factorize thevariational distribution in the following form:Q(H) =∏

i Qi(Hi), in a conjugate-exponential model1, Winn andBishop shows that the variational estimation for each nodecan be obtained by iteratively passing messages betweennodes in the network and updating posterior beliefs usinglocal operations [29].

Note the parents and children of nodei aspai andchi

respectively, we can obtain

lnQ∗i (Hi) = < lnP (Hi|pai) >∼Q(Hi) + (6)

∑

k∈chi

< lnP (Hk|pak) >∼Q(Hi) +const

which can be solved by passing variational message be-tween neighboring nodes [29].

Following the VMP framework, we can obtain the varia-tional estimation for nodezr as:

E[lnPr(zr = k)] = θk + ln Pr(ar|zr) +X

i

ln Pr(wir|zr), (7)

To update nodeλ, one can fit a Dirichlet distribution withθ by maximize likelihood estimation [21]. For other nodes,the update equations are

E[ln θk] = Ψ(γk) − Ψ(∑K

k=1 γk) (8)

E[lnβ] = Ψ(µw) − Ψ(∑W

w=1 µw) (9)

E[lnα] = Ψ(νa) − Ψ(∑A

a=1 νw) (10)

whereΨ is a digamma function and

γk = λk +∑Rd

r=1 δ(zr, k) (11)

µw =∑

zrNum(wr) + ηβ (12)

νa =∑

zrNum(ar) + ηα (13)

Here we omit the details due to the space limit. The readermay refer to [29] for detailed explanation.

3.2. Supervised Spatial-LTM

Till now we have been discussing how to learn theSpatial-LTM without supervision. As most generative mod-els, Spatial-LMT also enjoys the flexibility of handling bothlabeled and unlabeled data.

1the distributions of variables conditioned on their parents are drawnfrom the exponential family and are conjugate with respect to the distribu-tion over these parent variables.

There are two possible ways of introducing user super-vision. One way is to provide a category label for eachimage. Such supervision information is important for an-alyzing complex scenes [7]. For example, users can assigna “beach” or “kitchen” category label to one image. In thiscase, an additional categorical label is given to the entireimage (Fig.4(b)) as oppose to the object categorization casewhere latent topics for each region represent the categoricallabel (Fig.4(c)).

To incorporate this information of image categories, weadd a new nodec in the graphical model for each image.Each possible value ofc corresponds to a distribution overthe topicsθ, which now becomes aC × K matrix. Eachrow of this matrix denotes a Dirichlet distribution ofθ givencategoryc. Fig.4 (c) shows the graphical model representa-tion of Spatial-LTM with category labels. To learn the su-pervised Spatial-LTM, we can still use VMP for inference.Most of the inferring steps are similar with its unsupervisedcounterpart. Only the update equations for theθ node areslightly modified.

γck = Φ∗(θ)ck = δ(c − c∗)(λck − 1) +

RdX

r=1

δ(zr, k) + 1, (14)

E[Q∗] = E[ln θck|γ] = Ψ(γck) − Ψ(K

X

k=1

γck) (15)

To estimate the class label of each image, we choose thecategory label with the highest probability.

c∗ = arg max

c

Y

r∈Id

Pr(ar,wr|θc) (16)

Another way of introducing supervision is to providesome of the topic labels and position of the objects. Giventhe position of an object in one image, we can assign the ob-ject label to the corresponding topiczr if region r belongsto the given object. To learn the model with partially knownzr, we treat the nodes of the givenzr as visible nodes anddo not update the corresponding parameters.

4. Experimental Results

4.1. Unsupervised Spatial-LTM

We first apply the Spatial-LTM algorithm to automati-cally extract horses and cows, using the Weizmann dataset(327 horses, [4]) and the Microsoft object recognitiondataset (182 cows, [30]). Note that our method differsfrom [16] and [4] since both of their methods require hu-man labels of the training images. The only unsupervisedalgorithm on horse databases is LOCUS [31]. LOCUS is ashape-based method. It therefore requires all the horses inthe pictures to face the same direction. Spatial-LTM doesnot need to make such assumptions. The Microsoft cow is a

Figure 5.Segmentation and classification results of horses and cows.Theregions in red color are the segmentations of the animals. The regions ofother colors stand for three classes of backgrounds. The last row shows thatour method can find the object in inverted direction and undersignificantocclusion.

more challenging dataset, where the animals face three dif-ferent directions (left, right, front). In addition, some cowpictures also contain multiple instances and significant oc-clusions.

We obtain an average segmentation accuracy of81.8%on the horse dataset. This is measured by the percentageof pixels in agreement with the ground truth segmentation.We cannot measure quantitatively the segmentation resultsfor the cow images due to the lack of ground-truth of thisdataset. However, Fig.5 shows Spatial-LTM can success-fully recognize the cows facing different directions and evenunder significant occlusions. Moreover, we test our algo-rithm with images in which the horses are inverted or oc-cluded by vertical bars (last row in Fig.5). Our results showthat our algorithm can handle such situations. LOCUS onthe other hand cannot recognize horses facing opposite di-rections, or under such occlusions.

We next test the Spatial-LTM on the Caltech 5 dataset(four objects and one background). Fig.6 gives examples ofunsupervised segmentation results. Spatial-LTM also clas-sifies the images according to the object category they con-tain. This is done by selecting the most probable latent topic

Figure 6.Segmentation and classification results of the Caltech5 objectsdatabase. The four foreground classes of objects (airplane,car brad, motor-bike, and face) are shown in different color masks.

coast

co

ast

forest

fore

st

highway

hig

hw

ay

mountain

mo

un

tain

opencountry

op

en

co

un

try

bedroom

be

dro

om

kitchen

kitc

he

n

livingroom

livin

gro

om

PARoffice

PA

Ro

ffice

CALsuburb

CA

Ls

ub

urb

insidecity

ins

ide

city

street

stre

et

tallbuilding

tallb

uild

ing

85 2 2 1 10

98 2

16 2 66 6 6 1 2 1

2 23 8 61 2 1 2 1

12 25 7 3 50 3

1 2 1 2 34 23 4 18 2 2 6 5

2 49 3 29 14 1 2

1 1 6 19 37 17 2 8 5 4

11 86

4 1 3 1 1 5 83

4 2 1 6 2 4 3 70 3 5

1 1 5 2 1 1 3 10 72 4

2 6 6 1 3 1 6 5 70

Overall classification precision: 66.4%

Figure 7.Confusion matrix of classifying 13 classes of scene images.The rows denote true label and the columns denote estimated label. All thenumbers stand for the percentage number.

inferred from the images as in Eq.(3). We compare the clas-sification performance of Spatial-LTM and the pLSA modelused in [27]. Our performance is measured by the aver-age of classification precision for each class. The Spatial-LTM model obtains a classification precision of94.6%, bet-ter than the results obtained by the pLSA algorithm (83%)when performing the 5-way classification. In addition, ourmethod is capable of segmenting out the objects from theimages whereas pLSA [27] cannot do so (see examples inFig.6).

4.2. Supervised Spatial-LTM

We illustrate the results of Spatial-LTM with supervisedlearning in two different experiments. The first experimentuses the scene dataset [7], which contains 13 classes of na-ture scenes. For each category, we randomly select 100images with their category labels for training the model inFig.4(b). For this data set we set the number of topics to60,and the number of categories to13. In testing, we use thetrained model to concurrently classify and segment the im-ages. Fig.7 shows the confusion table of the classification

Figure 8.Segmentation results of the scene images[7]. The topics ofregions are represented by different colors, superimposedon the originalimage.

result. Spatial-LTM obtains an overall classification preci-sion of66.4%, comparable with the results in [7] (65.2%).We also conduct the segmentation for the scene images.Note that this task is harder compared with segmentationof the horses/cows database or Caltech5 database, since thescene images are cluttered with different types of objects.Fig. 7 shows some of the examples of the segmentation re-sults.

As discussed in Section3.2, Spatial-LTM can also trainwith partially labeled latent topics. In this experiment, weselect28 classes of objects among the subset of categoriesthat contain more than60 images from the Caltech101dataset [8] to illustrate our results. These categories con-tain objects with jagged boundaries (e.g. sunflower), thinregions (e.g. flamingo), or peripheral structures (e.g. cup).For each class, we randomly select30 images for trainingand30 images for testing. In training, we use the knowledgethat the foreground object in each category has a known as-signment (groundtruth mask annotated by human subjects).We choose28 topics for describing the object classes, and6more topics for the background. Note that only the28 fore-ground topics are labeled during training. The6 backgroundclasses are left unlabeled. Given a new image, the mostprobable latent topick is selected as the category label ofthe object. The union of regions wherez∗r = k provide thesegmentation result of the object given by Eq.(4). It takesless than 10 seconds to classify and segment a new image,including computing the local descriptors and preparing theregion-based over-segmentation.

To the best of our knowledge, we are the first one toprovide both the segmentation and classification results onnearly 30 classes of objects. Fig.9 and Fig.10 show theconfusion table of the classification and the segmentationaccuracy, respectively. The extensive segmentation resultsin Fig. 11and Fig.12validates the success of our method.

Faces easy

Faces e

asy

Leopards

Leo

pard

s

Motorbikes

Mo

torb

ikes

bonsai

bo

nsai

brain

bra

in

butterfly

bu

tterfly

cougar face

co

ug

ar fa

ce

crab

crab

cup

cu

p

dalmatian

dalm

atian

dolphin

do

lph

in

elephant

ele

ph

an

t

euphoniumeu

ph

on

ium

ewerew

er

ferry

ferry

flamingo

flam

ing

o

grand piano

gra

nd

pia

no

joshua tree

josh

ua tree

kangaroo

kan

garo

o

laptop

lap

top

lotus

lotus

schooner

sch

oo

ner

soccer ball

so

ccer b

al

l

starfish

sta

rfish

stop sign

sto

p s

ign

sunflower

su

nflo

wer

watch

watc

h

yin yang

yin

yan

g

9673 181 1 2 1 2 1 1 1 1 1

804 1 1 1 1 1 3 1 1 1 1 32 2721 1 1 5 1 2 1 1 2 1 1 5

821 3 1 1 1 13 1 1 313421 1 1 1 1 1 3 4 6 4 110

2 2 4 82 2 6 26 4 4 4 2 4452 2 2 4 4 2 4 6 2 4 2

3 3 3 3 703 3 3 52 2 2 91 2

2 7 2 845 2 2 2 564 2 5 2 5 2 2

7 2 5 863 2 8 6 2 5 2 3 2 6 26 2 2 2 2 2 3 3 615

4 2 2 912 911 2492 2 2 2 2 9

1912 2 5 2 2 2 277 53 2 9 6 2 2 3 2 2 3612 2 2 2 2

2 2 3 2 2 2 13 62 107 2 4 80 2 42 4 41 5

2 2 2 70 7 7 72 5 3 5 2 6 52 2 2 3 3 3 2 3 45 2 9

2 2 2 2 912 2 2 2 2 91 2

5 7 1 1 1 1 1 1 5 1 1 25885 3 3 3 88

Overall classification precision: 69.4%

Figure 9.Confusion matrix for the classification of 28 classes from theCaltech101 database [8]. Each row denotes ground-true label of the objectclass. And each column denotes class label predicted by the model. Allthe numbers stand for the percentage number. The overall classificationprecision is obtained by average the diagonal entries of theconfusion table.

Figure 10.Segmentation accuracy of for the 28 classes from the Cal-tech101 database [8]. The horizontal axis shows the abbreviated names ofeach class and the vertical axis represents the segmentation accuracy. Thethick bars stand for the average accuracy for each class, andthe thin barsare the standard deviation of the accuracy within each class.

5. Conclusion

In this paper, we propose a novel model Spatial-LTMfor object segmentation and classification. Our model aimsto close the gap between bag of words models and imageanalysis. By considering neighboring appearance of localdescriptors, Spatial-LTM exceeds the bag of words modelwith satisfactory segmentation ability as well as better clas-sification performance.

Acknowledgement

We would like to thank Silvio Savarese, Sinisa Todorovicand anonymous reviewers for helpful comments. L.F-F issupported by Microsoft Research New Faculty Fellowship.

joshua tree

joshua tree

joshua tree

lotus

lotus

lotus

flamingo

flamingo

flamingo

sunflower

sunflower

sunflower

Figure 11.Examples of the object segmentation results for some of the 28 classes from the Caltech101 database [8]. Note our method can also providethe class label of the segmented object. For each triplet of the examples, we first show the original testing image. The middle image shows the foregroundobject in white color and the background topics in black. Andthe last image shows the segmented object free of background.

Figure 12.More segmentation results for some of the Caltech101 objectclasses.

References[1] S. Belongie, J. Malik, and J. Puzicha. Shape matching andobject

recognition using shape contexts.PAMI, 24(4):509–522, 2002.2[2] D. Blei, A. Ng, and M. Jordan. Latent dirichlet allocation. Journal

of Machine Learning Research, 3:993–1022, 2003.1, 4[3] E. Borenstein, E. Sharon, and S. Ullman. Combining top-down and

bottom-up segmentation.CVPR Workshop, pages 46–46, 2004.3[4] E. Borenstein and S. Ullman. Learning to segment.ECCV, pages

315–328, 2004.2, 3, 5[5] P. Carbonetto, N. de Freitas, and K. Barnard. A statistical model for

general contextual object recognition.ECCV, 2004.2[6] G. Csurka, C. Bray, C. Dance, and L. Fan. Visual categorization with

bags of keypoints.Workshop on Statistical Learning in ComputerVision, ECCV, pages 1–22, 2004.1

[7] L. Fei-Fei and P. Perona. A bayesian hierarchical model for learningnatural scene categories.CVPR, pages 524–531, 2005.1, 5, 6, 7

[8] L. Fei-Fei and P. Perona. One-shot learning of object categories.PAMI, 28(4):594 – 611, 2006.7, 8

[9] P. Felzenszwalb and D. Huttenlocher. Efficient graph-based imagesegmentation.IJCV, 59(2):167–181, 2004.2, 3

[10] P. Felzenszwalb and D. Huttenlocher. Pictorial structures for objectrecognition.IJCV, 61(1):55–79, 2005.2

[11] R. Fergus, L. Fei-Fei, P. Perona, and A. Zisserman. Learning objectcategories from google’s image search.ICCV, 2005.1, 2

[12] R. Fergus, P. Perona, and A. Zisserman. Object class recognition byunsupervised scale-invariant learning.CVPR, 2:264–271, 2003.2

[13] T. Hofmann. Unsupervised learning by probabilistic latent semanticanalysis.Machine Learning, 42(1):177–196, 2001.1

[14] M. Jordan, Z. Ghahramani, T. Jaakkola, and L. Saul. An introduc-tion to variational methods for graphical models.Machine Learning,37(2):183–233, 1999.4

[15] T. Kadir and M. Brady. Scale, saliency and image description. Int’lJournal Computer Vision, 45(2):83–105, 2001.2, 3

[16] M. P. Kumar, P. H. S. Torr, and A. Zisserman. OBJ CUT.CVPR,pages 18–25, 2005.2, 3, 5

[17] B. Leibe, A. Leonardis, and B. Schiele. Combined objectcategoriza-tion and segmentation with an implicit shape model.ECCV Work-shop, 2004.2, 3

[18] B. Leibe and B. Schiele. Interleaved object categorization and seg-mentation.BMVC, 2003.2, 3

[19] A. Levin, Y. Weiss, and M. Vision. Learning to combine top-downand bottom-up segmentation.ECCV, 2006.3

[20] D. Liu and T. Chen. Semantic-shift for unsupervised object detection.CVPR workshop, 2006.1

[21] T. Minka. Estimating a dirichlet distribution.Technical report, Mi-crosoft Research. 5

[22] F. Monay and D. Gatica-Perez. PLSA-based image auto-annotation:constraining the latent space.ACM Conf. Multimedia, 2004.1

[23] D. Proc. Lowe. Object recognition from local scale-invariant fea-tures.Proc. Int’l Conf. Computer Vision, 1999.2, 3

[24] C. Rother, V. Kolmogorov, T. Minka, and A. Blake. Cosegmentationof image pairs by histogram matchingłincorporating a global con-straint into MRFs.CVPR, 2006.3

[25] B. Russell, A. Efros, J. Sivic, W. Freeman, and A. Zisserman. Us-ing Multiple Segmentations to Discover Objects and their Extent inImage Collections.CVPR, 2006.1, 2, 3

[26] C. Schmid. Constructing models for content-based image retrieval.CVPR, 2:39–45, 2001.3

[27] J. Sivic, B. Russell, A. Efros, A. Zisserman, and W. Freeman. Dis-covering object categories in image collections.ICCV, 1:65, 2005.1, 3, 6

[28] S. Todorovic and N. Ahuja. Extracting subimages of an unknowncategory from a set of images.CVPR, 2006.3

[29] J. Winn and C. Bishop. Variational message passing.Journal ofMachine Learning Research, 6:661–694, 2005.4, 5

[30] J. Winn, A. Criminisi, and T. Minka. Object categorization bylearned universal visual dictionary.ICCV, 2:1800 – 1807, 2005.5

[31] J. Winn and N. Jojic. LOCUS: Learning object classes with unsuper-vised segmentation.ICCV, 2005.2, 3, 5