SUBMITTED TO IEEE TRANSACTIONS ON PATTERN …fergus/drafts/pamipaper.pdfapproach learns hierarchical...

SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 1

Learning a Hierarchical Compositional ShapeVocabulary for Multi-class Object Representation

Sanja Fidler, Student Member, IEEE, Marko Boben, and Ales Leonardis, Member, IEEE

Abstract—Hierarchies allow feature sharing between objects at multiple levels of representation, can code exponential variability ina very compact way and enable fast inference. This makes them potentially suitable for learning and recognizing a higher number ofobject classes. However, the success of the hierarchical approaches so far has been hindered by the use of hand-crafted features orpredetermined grouping rules. This paper presents a novel framework for learning a hierarchical compositional shape vocabulary forrepresenting multiple object classes. The approach takes simple contour fragments and learns their frequent spatial configurations.These are recursively combined into increasingly more complex and class-specific shape compositions, each exerting a high degree ofshape variability. At the top-level of the vocabulary, the compositions are sufficiently large and complex to represent the whole shapesof the objects. We learn the vocabulary layer after layer, by gradually increasing the size of the window of analysis and reducing thespatial resolution at which the shape configurations are learned. The lower layers are learned jointly on images of all classes, whereasthe higher layers of the vocabulary are learned incrementally, by presenting the algorithm with one object class after another. Theexperimental results show that the learned multi-class object representation scales favorably with the number of object classes andachieves a state-of-the-art detection performance at both, faster inference as well as shorter training times.

Index Terms—Hierarchical representations, compositional hierarchies, unsupervised hierarchical structure learning, multiple objectclass recognition and detection, modeling object structure.

F

1 INTRODUCTION

V ISUAL categorization of objects has been an areaof active research in the vision community for

decades. Ultimately, the goal is to recognize and de-tect an increasing number of object classes in imageswithin an acceptable time frame. The problem entanglesthree highly interconnected issues: the internal objectrepresentation which should compactly capture the highvisual variability of objects and generalize well over eachclass, means of learning the representation from a setof images with as little supervision as possible, and aneffective inference algorithm that robustly matches theobject representation against the image.

Using vocabularies of visual features has been a pop-ular choice of object class representation and has yieldedsome of the most successful performances for objectdetection to date [1], [2], [3], [4], [5], [6]. However, themajority of these works are currently using flat codingschemes where each object is represented with either nostructure at all by using a bag-of-words model or onlysimple geometry induced over a set of intermediatelycomplex object parts. In this paper, our aim is to modelthe hierarchical compositional structure of the objects anddo so for multiple object classes.

Modeling structure (geometry of objects) is importantfor several reasons. Firstly, since objects within a classhave usually distinctive and similar shape, it allows for

• S. Fidler, M. Boben and A. Leonardis are with the Faculty of Computerand Information Science, University of Ljubljana, Slovenia.E-mail: sanja.fidler,marko.boben,[email protected]

an efficient shape parametrization with good general-ization capabilities. It further enables us to parse objectsinto meaningful components which is of particular im-portance in robotic applications where the task is notonly to detect objects but also to execute higher-levelcognitive tasks (manipulations, grasping, etc). Thirdly,by inducing the structure over the features the represen-tation becomes more robust to background clutter.

Hierarchies incorporate structural dependenciesamong the features at multiple levels: objects aredefined in terms of parts, which are further composedfrom a set of simpler constituents, etc [7], [8], [9],[10], [11]. Such architectures allow sharing of featuresbetween the visually similar as well as dissimilarclasses at multiple levels of specificity [12], [13], [11],[10]. Sharing of features means sharing commoncomputations and increasing the speed of the jointdetector [2]. More importantly, shared features lead tobetter generalization [2] and can play an important roleof regularization in learning of novel classes with fewtraining examples. Furthermore, since each feature inthe hierarchy recursively models certain variance overits parts, it captures a high structural variability andconsequently a smaller number of features are neededto represent each class.

Learning of feature vocabularies without or with littlesupervision is of primary importance in multi-class ob-ject representation. By learning, we minimize the amountof time of human involvement in object labeling [9],[14] and avoid bias of pre-determined grouping rules ormanually crafted features [15], [16], [17], [18]. Second,learning the representation statistically yields features


most shareable between the classes, which may notbe well predicted by human labelers [19]. However,the complexity of learning the structure of a hierarchicalrepresentation bottom-up and without supervision isenormous: there is a huge number of possible featurecombinations, the number of which exponentially in-creases with each additional layer — thus an effectivelearning algorithm must be employed.

In this paper, the idea is to represent the objects with alearned hierarchical compositional shape vocabulary that hasthe following architecture. The vocabulary at each layercontains a set of hierarchical deformable models whichwe will call compositions. Each composition is definedrecursively: it is a hierarchical generative probabilisticmodel that represents a geometric configuration of asmall number of parts which are themselves hierarchicaldeformable models, i.e., compositions from a previouslayer of the vocabulary. We present a framework forlearning such a representation for multiple object classes.Learning is statistical and is performed bottom-up. Theapproach takes simple oriented contour fragments andlearns their frequent spatial configurations. These arerecursively combined into increasingly more complexand class-specific shape compositions, each exerting ahigh degree of shape variability. In the top-level of thevocabulary, the compositions are sufficiently large andcomplex to represent the whole shapes of the objects.We learn the vocabulary layer after layer, by graduallyincreasing the size of the window of analysis and thespatial resolution at which the shape configurations arelearned. The lower layers are learned jointly on images ofall classes, whereas the higher layers of the vocabularyare learned incrementally, by presenting the algorithmwith one object class after another. We assume supervi-sion in terms of a positive and a validation set of classimages — however, the structure of the vocabulary islearned in an unsupervised manner. That is, the numberof compositions at each layer, the number of parts foreach of the compositions along with the distribution pa-rameters are inferred from the data without supervision.

We experimentally demonstrate several important is-sues: 1.) Applied to a collection of natural images, theapproach learns hierarchical generative models for vari-ous curvatures, T- and L- junctions, i.e., features usuallyemphasized by the Gestalt theory [20]; 2.) We show thatthese generic compositions can be effectively used forobject classification; 3.) For object detection we demon-strate a competitive speed of detection with respect tothe related approaches already for a single class. 4.) Formulti-class object detection we achieve a highly sub-linear growth in the size of the hierarchical vocabulary atmultiple layers and, consequently, a scalable complexityof inference as the number of modeled classes increases;5.) We demonstrate a competitive detection accuracywith respect to the current state-of-the-art. Furthermore,the learned representation is very compact — a hierarchymodeling 15 object classes uses only 1.6Mb on disk.

The remainder of this paper is organized as follows. In

Section 2 we review the related work. Section 3 presentsour hierarchical compositional representation of objectshape with recognition and detection described in Sec-tion 4. In Section 5 our learning framework is proposed.The experimental results are presented in Section 6. Thepaper concludes with a summary and discussion inSec. 7 and pointers to future work in Sec. 8.

2 RELATED WORK AND CONTRIBUTIONS

Compositional hierarchies. Several compositional ap-proaches to modeling objects have been proposed inthe literature, however, most of them relied on hand-crafted representations, pre-determined grouping rulesor supervised training with manually annotated objectparts [9], [16], [17], [21]. The reader is referred to [9],[22] for a thorough review.

Unsupervised learning of hierarchical compositionalvocabularies. Work on unsupervised hierarchical learninghas been relatively scarce. Utans [23] has been the firstto address unsupervised learning of compositional rep-resentations. The approach learned hierarchical mixturemodels of feature combinations, and was utilized onlearning simple dot patterns.

Based on the Fukushima’s model [24], Riesenhuberand Poggio [25] introduced the HMAX approach whichrepresents objects with a 2-layer hierarchy of Gaborfeature combinations. The original HMAX used a vo-cabulary of pre-determined features, while these havesubsequently been replaced with randomly chosen tem-plates [26]. Since no statistical learning is involved, asmuch as several thousands of features are needed to rep-resent the objects. An improved learning algorithm hasrecently been proposed by Masquelier and Thorpe [27].

Among the neural network representatives, Convo-lutional nets [28], [29] have been most widely andsuccessfully applied to generic object recognition. Theapproach builds a hierarchical feature extraction andclassification system with fast feed-forward processing.The hierarchy stacks one or several feature extractionstages, each of which consists of filter bank layer, non-linear transformation layers, and a pooling layer thatcombines filter responses over local neighborhoods us-ing an average or max operation, thereby achievinginvariance to small distortions [29]. A similar approachis proposed by Hinton [30] with recent improvementsby Ng et al. [31]. One of the main drawbacks of theseapproaches, however, may be that they do not explicitlymodel the spatial relations among the features and arethus less robust to shape variations.

Bouchard and Triggs [7] proposed a 3−layer hierarchy(extension of the constellation model [1]) and a similarrepresentation was proposed by Torralba et al. [2], [32].For tractability, all of these models are forced to usevery sparse image information, where prior to learningand detection, a small number (around 30) of interestpoints are detected. Using highly discriminative SIFTfeatures might limit their success in cluttered images or


on structurally simpler objects with little texture. Therepeatability of the SIFT features across the classes isalso questionable [26]. On the other hand, our approachis capable of dealing with several tens of thousands ofcontour fragments as input. We believe that the use ofrepeatable, dense and indistinctive contour fragmentsprovide us with a higher repeatability of the subsequentobject representation facilitating a better performance.

Epshtein and Ullman [33] approached the represen-tation from the opposite end; the hierarchy is built bydecomposing object relevant image patches into recur-sively smaller entities. The approach has been utilized onlearning each class individually while a joint multi-classrepresentation has not been pursued. A similar line ofwork was adopted by Mikolajczyk et al. [34] and appliedto recognize and detect multiple (5) object classes.

Todorovic and Ahuja [35] proposed a data-drivenapproach where a hierarchy for each object example isgenerated automatically by a segmentation algorithm.The largest repeatable subgraphs are learned to repre-sent the objects. Since bottom-up processes are usuallyunstable, exhaustive grouping is employed which resultsin long training and inference times.

Ommer and Buhmann [11] proposed an unsupervisedhierarchical learning approach, which has been success-fully utilized for object classification. The features at eachlayer are defined as histograms over a larger, spatiallyconstrained area. Our approach explicitly models thespatial relations among the features, which should al-low for a more reliable detection of objects with lowersensitivity to background clutter.

The learning frameworks most related to ours includethe work by [36], [37] and just recently [38]. However,all of these methods build separate hierarchies for eachobject class. This, on the one hand, avoids the massivenumber of possible feature combinations present in di-verse objects, but, on the downside, does not exploit theshareability of features among the classes.

Our approach is also related to the work on multi-classshape recognition by Amit et al. [39], [40], [13], and someof their ideas have also inspired our approach. Whilethe conceptual representation is similar to ours, thecompositions there are designed by hand, the hierarchyhas only three layers (edges, parts and objects), and theapplication is mostly targeted to reading licence plates.

While surely equally important, the work on learningof visual taxonomies [41], [42] of object classes tacklesthe categorization/recognition process by hierarchicalcascade of classifiers. Our hierarchy is compositional andgenerative with respect to object structure and does notaddress the taxonomic organization of object classes.

Contour-based recognition approaches. Since our ap-proach deals with object shape, we also briefly reviewthe related work on contour-based object class detection.

Contour fragments have been employed in the earlierwork by Selinger and Nelson [43] and a number offollow-up works have used a similar approach. Opeltet al. [3] learned boundary fragments in a boosting frame-

work and used them in a Hough voting scheme for ob-ject class detection. Ferrari et al. [44], [5] extracted kPASfeatures which were learned by combining k roughlystraight contour fragments. The learned kPAS featuresresemble those obtained in the lower hierarchical layersby our approach. Fergus et al. [45], [1] represent objectsas constellations of object parts and propose an approachthat learns the model from images without supervisionand in a scale invariant manner. The part-based repre-sentation is similar to ours, however in contrast, ourapproach learns a hierarchical part-based representation.

To the best of our knowledge, we are the first to1.) learn generic shape structures from images at mul-tiple hierarchical layers and without supervision, which,interestingly, resemble those predicted by the Gestalttheory [20], 2.) learn a multi-class generative composi-tional representation in a bottom-up manner from simplecontour fragments, 3.) demonstrate scalability of a hier-archical generative approach when the number of classesincreases — in terms of the speed of inference, storage,and training times.

This article conjoins and extends several of our confer-ence papers on learning of a hierarchical compositionalvocabulary of object shape [46], [10], [47], [48]. Specif-ically, the original model has been reformulated into aprobabilistic form and more details have been includedwhich could not be given in the conference papers due topage constraints. Further, more classes have been used inthe experiments, the approach has been tested on severalstandard recognition benchmarks and an analysis withrespect to the computational behavior of the proposedapproach had been carried out.

3 A HIERARCHICAL COMPOSITIONAL OBJECTREPRESENTATION

The complete hierarchical framework addresses threemajor issues: the representation, inference and learning.In this section we present our hierarchical compositionalobject representation.

Our aim is to model the distribution of object shapesin a given class and we wish to do so for multipleobject classes in a computationally efficient way. Theidea is to represent the objects with a learned hierarchi-cal compositional shape vocabulary that has the followingarchitecture. The vocabulary at each layer contains a setof hierarchical deformable models which we will callcompositions. Each composition is defined recursively:it is a hierarchical generative probabilistic model thatrepresents a geometric configuration of a small numberof parts which are themselves hierarchical probabilisticmodels, i.e., compositions from a previous layer of thevocabulary. Different compositions can share models forthe parts, which makes the vocabulary efficient in sizeand results in faster inference. We will set up our learn-ing algorithms in a way that, during inference, differentcompositions at a particular layer of the vocabulary willcompete for the explanation of an input image. That is,


Fig. 1. Left: Illustration of the hierarchical vocabulary. Theshapes depict the compositions ω` and the links between themdenote the compositional relations between them. Note thatthere are no actual images stored in the vocabulary – thedepicted shapes are the samples from the generative models.Right: An example of a complete composition model from layer3. The blue patches are the limiting sizes within which themodel can generate shapes, while the red ellipses denote theGaussians representing the spatial relations between the parts.

each patch in an image will, in most cases, be explainedby only one of the compositions in the vocabulary.

Specifically, each part in a composition has an “ap-pearance” which is defined as a (discrete) distributionover the set of compositions from the previous layer ofthe vocabulary. The geometric configuration of parts ismodeled by relative spatial relations between each of theparts and one part called a reference part. The hierarchicaltopology of a composition is assumed to be a tree (andwill be learned in a way to guarantee this), meaning thatnone of its parts spatially overlap. Different composi-tions can have different topologies (different tree struc-tures) which will be learned in an unsupervised mannerfrom images, along with the corresponding parameters.The number of compositions that make a particular layerin the vocabulary will also not be set in advance butdetermined through learning. We will, however, assumesupervision on the object level, i.e., the set of positiveand validation images of objects in each class needs tobe given. The amount of supervision the approach needsis further discussed in Section 5.3.

At the lowest (first) layer, the hierarchical vocabularyconsists of a small number of base models that representshort contour fragments at coarsely defined orientations.The number of orientations is assumed given. At the top-most layer the compositions will represent the shapes ofthe whole objects. For a particular object class, a smallnumber of top-layer compositions will be used to rep-resent the whole distribution of shapes in the class, i.e.,each top-layer composition will model a certain aspect

(3D view or an articulation) of the objects within theclass. The left side of Fig. 1 illustrates the representation.

Due to the recursive definition, a composition at anylayer of the vocabulary is thus an “autonomous” de-formable model. As such, a composition at the top layercan act as a detector for an object class, while a com-position at an intermediate layer acts as a detector fora less complex shape (for example, an L junction). Sincewe are combining deformable models as we go up in thehierarchy, we are capturing an increasing variability intothe representation while at the same time preserving thespatial arrangements of the shape components.

We will use the following notation. Let Ω denote theset and structure of all compositions in the vocabularyand Θ their parameters. Since the vocabulary has ahierarchical structure we will write it as Ω = Ω1 ∪ Ω2 ∪· · · ∪ΩO ∪ΩC where Ω` = ω`ii is a set of compositionsat layer `. Whenever it is clear from the context, we willomit the index i. With ΩO we denote the top, object layerof compositions (we will useO = 6 in this paper, and thischoice is discussed in Section 5.3), which roughly codethe whole shapes of the objects. The final, object classlayer ΩC is not compositional, but only pools all of thecorresponding object layer compositions for each classseparately. Specifically, a model for a particular objectclass c ∈ ΩC is represented as a disjunction (OR) of a(sub)set of object layer compositions.

The definition of a composition with respect to just onelayer below is akin to that of the constellation model [45].A composition ω`, where ` > 1, consists of P parts (P canbe different for different compositions) with appearancesθapp

` = [θapp`j ]Pj=1 and geometric parameters θg

` =[θg`j ]

Pj=1. The appearance θapp

`j of a part is a discrete

distribution over the compositions from the previouslayer. For example, θapp`j(ω

`−1k ) = 0.8 means that the j-th

part of ω` is 0.8 likely to be the k-th composition ω`−1k

from layer ` − 1. The geometric relations between theposition of each of the parts relative to the reference partare modeled by two-dimensional Gaussians with param-eters θg`j = (µ`j ,Σ

`j). For the convenience of notation later

on, we will also represent the location of a reference partwith respect to itself with a Gaussian having zero meanand small variance, ε2Id.

The models at the first layer of the vocabulary aredefined over the space of image features, which willin this paper be n-dimensional Gabor feature vectors(explained in Sec. 4.1). Each model ω1

i has an appearancedistribution over the feature vectors, which is taken tobe an n-dimensional Gaussian with parameters (µ1

i ,Σ1i ).

Lastly, the class layer models are only specified byan appearance distribution over the object layer com-positions. A class model c thus has the appearancedistribution θapp

Cc , which is non-zero only for those

object layer compositions that represent the object shapesin class c. We will denote this set by ωOicic ⊂ ΩO, withθapp

Cc (ωOic ) > 0 for each ic and θapp

Cc (ωOi′c ) = 0 for all

other i′c. The sets of object layer compositions for two


Layer 1 Layer 2 Layer 3

Layer 4 Layer 5 Layer 6

Fig. 2. The figure shows inferred parse trees (Section 4.2)for an image for compositions from different layers ofthe hierarchy. The complexity and the size of shapesexplained by the compositions gradually increases withthe level of hierarchy.

different classes are always disjunct.Note that each composition can only generate shapes

within a limited spatial extent, that is, the windowaround a generated shape is of limited size and the sizeis the same for all compositions at a particular layer ` ofthe vocabulary. We will denote it with r`. The limitingsize r` increases exponentially with the level of hierarchy.Figure 2 depicts the gradual increase in complexity andsize of the compositions in the hierarchy.

4 RECOGNITION AND DETECTION

The model is best explained by first considering infer-ence. Thus, let us for now assume that the representationis already known and we are only concerned with rec-ognizing and detecting objects in a query image giventhe model. How we learn the representation will beexplained in Sec. 5.

Subsec. 4.1 explains the image features we use, whichform a set of observations, and how we extract themfrom an image. Subsec. 4.2 describes the inference of thehidden states of the model, while Subsec. 4.2.3 showshow to perform it in a computationally efficient way. InSubsec. 4.3, we explain how we detect objects given theinferred hidden states.

4.1 Extracting image featuresLet I denote a query image. The features which weextract from I should depend on how we define thebase models at the first layer Ω1 of the vocabulary. Inthis paper we choose to use oriented edge fragments,however, the learning and the recognition procedures aregeneral and independent of this particular choice.

In order to detect oriented edges in an image we usea Gabor filter bank:

gλ,ϕ,γ,σ(x, y, ψ) = e−u2+γ2v2

2σ2 cos(2πuλ

+ ϕ)

u = x cosψ − y sinψ, v = x sinψ + y cosψ,

where (x, y) represents the center of the filter’s domain,and the parameters in this paper are set to (λ, γ, σ) =

orient. 1orient. 2orient. 3orient. 4orient. 5orient. 6

Fig. 3. Left: Original image. Middle: Maximum overorientation energies, arg maxψ E(x, y, ψ). Right: Pointsin which n-dimensional Gabor feature vectors f are ex-tracted. Different colors denote different dominant orien-tations in features f .

(6, 0.75, 2). A set of two filter banks is used, one witheven, ϕ = 0, and the other with odd, ϕ = −π2 ,Gabor kernels defined for n equidistant orientations,ψ = iπn , i = 0, 1, . . . , n − 1. For the experiments in thispaper we use n = 6.

We convolve the image I with both filter banks andcompute the total energy for each of the orientations [49]:

E(x, y, ψ) =√r20(x, y, ψ) + r2

−π/2(x, y, ψ), (1)

where r0(x, y, ψ) and r−π/2(x, y, ψ) are the convolutionoutputs of even and odd Gabor filters at location (x, y)and orientation ψ, respectively. We normalize E to havethe highest value in the image equal to 1. We furtherperform a non-maxima suppression over the total energyE to find the locations of the local maxima for eachof the orientations. This is similar to performing theCanny operator and taking the locations of the binaryresponses. At each of these locations (x, y), the set ofwhich will be denoted with X = (x, y), we extractGabor features f = f(x, y) which are n-dimensionalvectors containing the orientation energy values in (x, y),f(x, y) =

[E(x, y, iπn )

]n−1

i=0. Specifically, the feature set F

is the set of all feature vectors extracted in the image,F = f(x, y), (x, y) ∈ X. The number of features inF is usually around 104 for an image of average sizeand texture. The image features (F,X) serve as theobservations to the recognition procedure. Figure 3 showsan example of feature extraction.

Scale. The feature extraction as well as the recognitionprocess is performed at several scales of an image: weuse a Gaussian pyramid with two scales per octave andprocess each scale separately. That is, feature extractionand recognition up to the object level O are performedat every scale and independently of other scales. Fordetection of object classes we then consider informationfrom all scales. Not to overload the notation, we will justrefer to one scale of I .

4.2 Inference

Performing inference in a query image with a givenvocabulary entails inferring a hierarchy of hidden statesfrom the observations (which are the image features(F,X)). The hidden states at the first layer will be theonly ones receiving input directly from the observations,


whereas all the other states will receive input from ahidden layer below.

Each hidden state z` has two variables z` = (ω`, x`),where ω` denotes a composition from the vocabularyand x` a location in an image (to abbreviate notationfrom here on, we will use x instead of (x, y) to denotethe location vector). Note that ω` is in fact a hierarchicalprobabilistic model, so the first variable is just a referenceto the model. The notation x` is used to emphasize thatx is from a spatial grid with resolution corresponding tohidden layer `, i.e., the spatial resolution of the locationsof the hidden states will be increasingly coarser with thelevel of hierarchy (discussed in Sec. 4.2.3).

We address three problems:1) Image likelihood. Calculating the likelihood of an

observed image neighborhood J under a particularcomposition (hierarchical model), say ω`: p(J |ω`).

2) Most probable hidden state activation. Findingthe most likely tree T ∗(J, ω`) of hidden state ac-tivations to have generated an observed imageneighborhood J given the model ω`: T ∗(J, ω`) =arg maxT p(T |J ;ω`).

3) Multi-class object detection and recognition.Finding the interpretation of an image in terms ofobject classes:

a) Finding the neighborhoods J of objects andthe class c they belong to: (Ji, ci) : p(Ji|ci) >τci

and Ji ∩ Jk = ∅; ∀ (i, k), i 6= k, whereτci

is a class-specific threshold that separatesforeground (class c) against the background.

b) Finding object segmentations which will bedefined by the leaves (the observations) of themost probable hidden state tree activationsT ∗(Ji, ci) for the detected objects (Ji, ci).

4.2.1 Calculating the likelihoodSince we will deal with images of much larger size thanr`, it is convenient to define I(x`) to be a neighborhood inan image I , centered in x` and radius r`,

I(x`) = (f , x) ∈ (F,X) : ||x− x`||2 ≤ r` (2)

A neighborhood I(x`) is thus defined as a collection offeatures in a circular region around x`.

For a given composition, e.g. ω`, we would like tocalculate the likelihood p(I(x`)|ω`) of an observed imageneighborhood I(x`) under ω`. By denoting z` = (ω`, x`)as a hidden state centered in x` (in the center ofthe neighborhood), then the following formulations areequivalent and will be used interchangeably throughoutthe paper: p(I(x`)|ω`) = p(I(x`), z`|ω`) = p(I(x`)|z`).Next, let T (z`) denote a tree of hidden states where theroot (top) node is z` and the leaves (layer 0) are theobservations. To calculate the likelihood, we thus needto sum over all possible such trees:

p(I(x`)|ω`) =∑T (z`)

p(I(x`), T (z`)|ω`) (3)

Since there is an exponential number of possible treesof hidden states, calculating this expression directlyis intractable. However, we can exploit the recursivedefinition of the compositions to get also a recursiveformulation for the likelihood.

We first write the joint distribution. Let T `′ denotethe set of states at layer `′ of the tree T (z`), let T (z`

′) be

the subtree of T (z`) that has the root z`′ ∈ T `′ , and let

T `−1 = [z`−1j := (ω`−1

j , x`−1j ))]|T

`−1|j=1 be the hidden states

at layer `−1 of the tree (just one layer below the root). Wewill re-write the joint p(I(x`), T (z`)|ω`) into a recursiveformulation. We first factorize it in the following way

p(I(x`), T (z`)|ω`) =P∏j=1

pF (I(x`), T (z`−1j ), z`−1

j |z`)︸︷︷︸foreground under part j

(4)

·|T `−1|∏k=P+1

pB(I(x`), T (z`−1k ), z`−1

k |z`)︸︷︷︸background

(5)

The foreground term can be further factorized:

pF (I(x`), T (z`−1j ), z`−1

j |z`) =pF (z`−1j |z`)· (6)

pF (I(x`), T (z`−1j )|z`−1

j ;ω`)

Since conditioning on layer ` − 1 renders lower layersindependent of `, the second term becomes

pF (I(x`), T (z`−1j )|z`−1

j ;ω`) = p(I(x`−1j ), T (z`−1

j )|ω`−1j )

(7)Given how we defined the compositions, the termpF (z`−1

j |z`) factorizes into appearance and geometry:

pF (z`−1j |z`) = p(ω`−1

j | θapp`j)︸︷︷︸appearance

p(x`−1j | x`; θg`j)︸︷︷︸

geometry

= θapp`j(ω

`−1j )N (x`−1

j − x` | µ`j ,Σ`j) (8)

where the first term is the probability of j-th part underω` having the appearance ω`−1

j (see Sec. 3). The secondterm is the spatial compatibility between the location ofstate z`−1

j and location x` of the parent z` under themodel for part j.

Similarly as (6), the background term in (4) factorizes:

pB(I(x`),T (z`−1k ), z`−1

k |ω`)= pB(z`−1

k |z`) pB(I(x`−1k ), T (z`−1

k )|z`−1k ;ω`)

= α ·(1− p(I(x`−1

k ), T (z`−1k )|ω`−1

k )), (9)

where the probability pB(z`−1k |z`) that a state z`−1

k be-longs to the background is taken to be α, and thelikelihood of the observations under a background dis-tribution is taken to be 1− the likelihood of the sameobservation to be the foreground.


The joint distribution in (4) can now be rewritten in arecursive form:

p(I(x`), T |ω`) =

=( P∏j=1

θapp`j(ω

`−1j )N (x`−1

j − x` | µ`j ,Σ`j)

· p(I(x`−1j ), T (z`−1

j )|ω`−1j )

)·(αNB

|T `−1|∏k=P+1

(1− p(I(x`−1

k ), T (z`−1k )|ω`−1

k ))), (10)

where NB = |T `−1| − P is the number of hidden statesin T `−1 belonging to the background.

Due to the recursive formulation of the joint, we cannow also get a recursive formulation for the likelihoodin (3). Let us define q(I, z) = p(I(x)|ω), where z = (ω, x).By manipulating the sums in (3) and (10), we obtain thefollowing form for the likelihood:

q(I, z`) =∑T `−1

[( P∏j=1

θapp`j(ω

`−1j )N (x`−1

j − x` | µ`j ,Σ`j)

· q(I, z`−1j )

)· αNB

|T `−1|∏k=P+1

(1− q(I, z`−1k ))

](11)

To complete the recursion we only need to define thedata term:

q(I, z1) = p(I(x1) | z1) = p(f(x1) | z1) = N (f | µ1,Σ1)(12)

q(I, z1) is a likelihood of a feature vector f (which isin our case a 6-dimensional vector of normalized Gaborfilter outputs) to be generated with a model ω1 for aparticular edge orientation. It is modeled with a (6-dimensional) Gaussian distribution.

The likelihoods can thus be computed efficiently bydynamic programming, similarly as in the forward algo-rithm for Hierarchical HMM [50]. This procedure alongwith further speed-ups inspired by the coarse-to-finesearch of Amit and Geman et al. [39], [37], [51] isdescribed in Subsec. 4.2.3.

4.2.2 Finding most probable hidden state activationGiven a model ω` we would additionally like to find themost probable tree of hidden states that generated animage neighborhood I(x`):

T ∗(z`) = arg maxT (z`)

p(T (z`) | I(x`);ω`)

= arg maxT (z`)

p(I(x`), T (z`) |ω`) (13)

If we define δ(I, z) = maxT (z) p(I(x), T (z)|ω), we canwrite a recursion for δ(I, z`) similar to that in (11),however, with max instead of the sum. For the data termwe have: δ(I, z1) = p(f(x1) | z1). The tree T ∗(z`) canthen be found by dynamic programming, similarly as inthe Viterbi recurrence for the Hierarchical HMM [50]: wecompute δ(I, z`) sequentially, from bottom layer to top

and for each hidden state we keep back-pointers to thestates at the previous layer that produced the maximum.The complete tree T ∗(z`) can then be found by tracingback the back-pointers in the resulting graph.

We will refer to the leaves of the tree T ∗(z`) as thesupport of a hidden state z`, and denote it with supp(z`).

4.2.3 Efficient computationThis section explains how to calculate the likelihoodsand the most probable trees efficiently. We will employthe following approach:• Recursive bottom-up computation: Compute the likeli-

hoods of the hidden states in a recursive bottom-upfashion, where the intermediate computations areshared among higher layer hidden states.

• Coarse-to-fine computation: Prior to computing thelikelihood of a state we will first perform a learnedcomputationally inexpensive test upon which wewill decide whether the likelihood of that state willbe computed or whether the state will be pruned.

Since ultimately, we want to recognize and detectobjects in images, our goal is to compute the image likeli-hoods under each of the object-layer compositions ωOi and do so for all image neighborhoods I(xOj ) (we willshow how the classes of the objects can be inferred basedon these likelihoods in Sec. 4.3). In order to computethe likelihood for a particular state zOij = (ωOi , x

Oj ),

the recursive formulation in (11) implies a bottom-upcomputation: start by computing the likelihoods of thehidden states at the first layer and continuing up. Notethat we do not need to compute the likelihoods of theintermediate hidden states for each zOij separately, butcan compute each of them only once (different statesdefined over overlapping neighborhoods can re-use theintermediate hidden states).

Notice, however, that this entails computing the like-lihoods q(I, z`) for all possible locations x` over theimage and all compositions ω` from the vocabulary, andfor each of the layers ` = 1, . . . ,O, which would stillcause inference to run slow. To make the computationsefficient, we will exploit the fact that most hidden stateswill be highly unlikely. We will make use of an ideaof coarse-to-fine search proposed by Amit and Gemanet al. [39], [37], [51]. The strategy will be to prune theunpromising hidden states based on computationallyinexpensive tests. A test is a simple decision functionemployed prior to computing the likelihood of a state,which tells us whether the state is worth consideringat all. In the case the test will return a positive answer,the likelihood for a particular state will be computed,while otherwise the state will be pruned. Note that dueto pruning, the resulting likelihoods in the hierarchywill not be exact. However, since only the very unlikelystates will be pruned, the computed likelihoods willrepresent a sufficiently good approximation. We explainthe complete procedure next.

Let G = G(I) = (Z,E) denote an inference graph, wherethe nodes Z of the graph are the (“promising”) hidden


states z`. With each state z` we also associate two values:the likelihood q(I, z`) and the probability of the mostprobable hidden tree δ(I, z`). Graph G has a hierarchicalstructure where the vertices Z are partitioned into vertexlayers 1 to O, that is, Z = Z0 ∪Z1 ∪ · · · ∪ZO. The vertexlayer Z` contains the hidden states z` from layer `, whileZ0 contains the observations (F,X) (each (f , x) ∈ (F,X)is a node in Z0).

Graph G is built from bottom (layer 1) to top (layer O).For a hidden state z` = (ω`, x`) we first perform a simplelearned test Tω`(I, z`) that has the following form:

Tω`(I, z`) = 1 (δ(I, z`) > τω`), (14)

where 1 is an indicator function and τω` is a learnedthreshold. A notation τω` is used, indicating that eachcomposition ω` in the vocabulary can have its ownthreshold. If T returns a positive answer then we addthe hidden state z` to Z` and calculate its likelihoodq(I, z`). In the case when T = 0, the state z` is notconsidered further, i.e., not added to the inference graph.Intuitively, a hidden state is pruned if only if even themost probable hidden tree for that state has a verysmall probability. During the process of training (untilthe hierarchy is built up to the level of objects), thesethresholds are fixed and set to very small values (in thispaper we use τ = 0.05). Their only role is to prunethe very unlikely hypotheses and thus both, speed-upcomputation as well as minimize the memory/storagerequirements (the size of G). After the class models (thefinal, layer ΩC of the vocabulary) are learned we can alsolearn the thresholds in a way to optimize the speed ofinference while retaining the detection accuracy, as willbe explained in Sec. 5.3.1.

For each state added to Z` we additionally form edgesin E from the state node z` to the states T `−1

∗ (z`) =[z`−1j ] ∈ Z`−1 that produced the value δ(I, z`).Reduction in spatial resolution. After each layer Z` is

built, we perform spatial contraction, where the locationsx` of the hidden states z` are downsampled by a factorρ` < 1. Among the states that code the same compositionω` and for which the locations (taken to have integervalues) become equal, only the state with the highestlikelihood is kept. We use ρ1 = 1 and ρ` = 0.5, ` > 1.This step is mainly meant to reduce the computationalload: by reducing the spatial resolution at each layer, webring the far-away (location-wise) hidden states closer.This, indirectly (through learning), keeps the scales ofthe Gaussians relatively small across all compositions inthe vocabulary and makes inference faster.

4.3 Object class detection and recognitionThis section explains how to perform

1) multi-class object recognition and detection,2) segmentation and parsing of the detected objects

once we have obtained (by using the procedures de-scribed in the previous sections) the inference graph Gbuilt to hidden layer ZO.

We first show how to compute the state likelihoodsunder the class models from the vocabulary (layer ΩC)given G. Based on these likelihoods we make a decisionon the presence of an object at a certain location. Anobject detection will, in this paper, taken to be a hiddenstate zC = (c, xC) with likelihood q(I, zC) higher thana class-specific threshold, that is, a threshold is used todecide whether a certain state belongs to the foreground(object) or background. Once this simple decision ismade we could additionally use a more sophisticatedverification step, e.g., employing SVM over the featuresin the detected region (as in [52], for example), however,we have not employed one in this paper.

Let ZC be a “class” hidden layer of G and let zC =(c, xC) ∈ ZC be a hidden state that represents the pres-ence of an object class c at location xC . Following (11),we can compute its likelihood q(I, zC) as follows:

q(I, zC) = p(I(xC) | zC) =∑

zOi

=(ωOi,xC)

p(zOi | zC) q(I, zOi )

=∑zO

i

θappCc (ωOi ) q(I, zOi ) (15)

Note that the sum runs over all possible states zOi atlocation xC . However, since θapp

Cc is zero for all but a

subset of layer O compositions, i.e., ωOic, we only needto sum over the corresponding hidden states zOic .

For object detection, we will only keep those states zC

that have likelihoods higher than a threshold, q(I, zC) >τc. Note that τc is a class-specific threshold, meaning thatdifferent decisions are made for different classes. Thehidden states not satisfying the decision inequality areremoved from ZC .

However, all nodes from ZC do not yet represent thefinal object detections, but define a set of object hypotheses.Since different neighborhoods in an image were assumedindependent during inference, we can potentially havemultiple different hypotheses at different locations inZC explaining partly the same image area, and thus aselection between the hypotheses is necessary. This isillustrated in Fig. 4: the left side shows the supports ofobject hypotheses for the horse class and the right sidefor the giraffe class. Ultimately, we would like to selectthe giraffe detection over the horse one, since it explainsa larger portion of the image, even though its likelihoodmight be somewhat lower than that under the horsehypothesis. For this reason, we will perform competitionamong overlapping hypotheses as explained below.

Competition between overlapping hypotheses. Let zCi andzCj denote two object hypotheses with overlapping sup-ports. To choose between them, we evaluate both hy-potheses on the union of their supports, i.e., supp(zCi ) ∪supp(zCj ). The unexplained features under each of thehypotheses count as background. We define the cost ofa hypothesis with respect to the other as

C(zCi |zCj ) =| supp(zCi )| · log q(I, zCi )+

| supp(zCj ) \ supp(zCi )| · logα. (16)


Fig. 4. Left: The supports of object hypotheses for classhorse. Right: Same for giraffe. See text for details.

The cost weights each feature in the support of ahypothesis zCi by the log-likelihood q(I, zCi ) and thebackground features (the features in supp(zCi )∪supp(zCj )not explained by zCi ) by logα. The hypothesis with thelower cost is selected as an object detection. The selectionprocess is run in a greedy fashion until all overlappinghypotheses are accounted for.

A parse tree (most probable tree of hidden states) foreach of the object detections zC is obtained by tracingback the edges in G, starting from the root node zC

down. A few examples of parse trees are shown in Fig. 15in Sec. 6. The segmentation of the object is defined by theleaves of the tree, i.e., the support of zC .

5 LEARNINGThis section addresses the problem of learning the rep-resentation, which entails the following:

1) Learning the vocabulary of compositions. Findingan appropriate set of compositions to representour data and learning the structure of each of thecompositions (the number of parts and a (rough)initialization of the parameters).

2) Learning the parameters of the representation.Finding the parameters of geometry and appear-ance for each composition as well as the thresholdsto be used in the approximate inference algorithm.

The objective of learning will be to maximize thelikelihood of the data. Since during recognition we areinterested in finding the compositions that best explainthe image in terms of likelihood, this objective is wellsuited for learning. Not to overfit the data, however,we will instead use a likelihood with a penalty for thecomplexity of the representation. Further, due to theexponential complexity of the unsupervised structurelearning problem, our strategy will be to learn one layerat a time, in a bottom-up fashion. Specifically, at eachstep, we will assume that layers from 1 to ` − 1 havebeen learned and are held fixed during learning of layer`. The problem thus reduces to learning a vocabulary atlayer ` by maximizing the penalized log-likelihood:

L(D; Ω`,Θ`) =N∑k=1

∑i

log p(Ik(xì) | Ω`,Θ`)− λ` |Ω`|

=N∑k=1

∑i

log∑Tki,ω`

ki

p(Ik(xì), Tki, ω`ki | Ω`Ω`,Θ`)−λ` |Ω`|,

(17)

where λ` is a layer-specific penalty for the complexityof the vocabulary. The first sum runs over all N trainingimages (which are the training data D), while the secondsum runs over all neighborhoods in an image. Fromhere on, we will omit the second sum for simplicity andassume that each training sample is an image neighbor-hood of appropriate size. The second term is a penaltyterm, which corresponds to the sum of the number ofparts across the compositions at layer `: |Ω`| =

∑ω` Pω` .

The sum runs over all compositions at layer ` of thevocabulary and Pω` denotes the number of parts that ω`

has. We will thus prefer structurally simpler composi-tions with high repeatability and descriptive power.

We will learn the vocabulary with a MCMC approach.Specifically, the objective in (17) will be used as a scoringfunction to compare different proposals for the vocab-ularies. At each step of the MCMC iteration, we willlearn a proposal vocabulary at layer `, use EM to learnthe parameters and calculate the score of the learnedvocabulary. Finally, the vocabulary with the highest scorewill be selected.

We will first assume that the structure of the repre-sentation is already known and show how to estimatethe parameters using the EM algorithm. In Sec. 5.2 wepropose how to learn also the structure of the vocabulary.

5.1 Learning the parametersLet us for now assume that the structure (the numberand the structure of the compositions) up to layer ` isgiven. We can employ the EM algorithm to estimate theparameters for each of the compositions at layer ` of thevocabulary. Since the model from one layer to the nextis similar to the constellation model [53], we will adoptsome of their derivations.

Instead of directly maximizing the likelihood in (17),EM iteratively re-estimates the parameters by maximiz-ing the cost functions:

Q(Θ`|Θ`) =∑k

E[

log p(Ik(x`k), Tk, ω`k|Θ`)]

(18)

Following [53], differentiating Q with respect to each ofthe parameters and setting the derivatives to 0 gives thefollowing updates for the parameters, which constitutethe M step of the EM algorithm:

µω`,j =∑k p(ω

`|Ik)Eω`,j [qk]∑k p(ω`|Ik)

Σω`,j =∑k p(ω

`|Ik)Eω`,j

[qk q

Tk

]∑k p(ω`|Ik)

− µω`,j µTω`,j

θàpp j(ω`−1) =

∑k p(ω

`|Ik)Eω`,j [δω`−1,ω`−1 ]∑k p(ω`|Ik)

p(ω`) =1N

∑k

p(ω`|Ik)

where qk = x`kj − x`k and δ here denotes the Kroneckerdelta. The expectation Eω` [·] is taken with respect to theposterior density p(Tk|Ik, ω`,Θ`).


In the E-step we need to compute the sufficient statis-tics Eω`,j [q], Eω`,j [q qT ] and the posterior density

p(ω`|Ik) =p(Ik|ω`)p(ω`)∑i p(Ik|ωì )p(ωì )

, (19)

where the likelihood p(Ik|ω`) is calculated as describedin Sec. 4.2.

We only still need to infer the parameters for thefirst layer models in the vocabulary. This is done byestimating the parameters (µ1

i ,Σ1i ) of a multivariate

Gaussian distribution for each model ω1i : Each Gabor

feature vector f is first normalized to have the dominantorientation equal to 1. All features f that have the i-th dimension equal to 1 are then used as the trainingexamples for estimating the parameters (µ1

i ,Σ1i ).

5.2 Learning the structureHere we explain how we learn the proposal set ofcompositions, their structure and rough initializations ofthe parameters (which is important for initializing EM).

We will assume that for each training image I we havethe inference graph GI = (Z1 ∪ · · · ∪Z`−1, E) built up tolayer `−1, which can be obtained as described in Sec. 4.2.

Our learning strategy will be the following:1) First learn the geometry distributions between all

possible pairs of compositions from layer `− 1.2) Detect the modes in these distributions which will

define two-part compositions called duplets.3) Find a set of compositions, where each composition

is a set of (frequently) co-occurring duplets. Theset (vocabulary) will be learned by the MCMCapproach, i.e., by stochastically searching for thevocabulary with the optimal score (17).

The overall learning procedure is illustrated in Fig. 5.

5.2.1 Learning spatial correlations between partsWe commence by learning the spatial constraints amongthe pairs of compositions (ω`−1

i , ω`−1j ) ∈ Ω`−1 × Ω`−1.

The first composition ω`−1i in the pair plays the role of

a reference. This means that the spatial constraints willbe learned relative to it. We further limit the relativedistances between the composition pairs to be at most r`

(the choice of r` is described below). We learn the spatialconstraints by means of two-dimensional histogramshìj : [−r`, r`]2 → R.

During training, each histogram hìj is updated atx`−1 − x`−1 for each pair of hidden states (z`−1, z`−1),where z`−1 = (ω`−1

i , x`−1) and z`−1 = (ω`−1j , x`−1),

satisfying:1) |x`−1 − x`−1| ≤ r`, that is, z`−1 lies in the circular

area around z with radius r`.2) z`−1 and z`−1 have sufficiently disjoint supports.

This is due to our assumption that the com-positions have a tree topology, i.e. none of theparts in each composition spatially overlap (orrather, the probability that two parts overlap is

small). The overlap of the supports is calculatedas o(z`−1, z`−1) = | supp(z`−1)∩supp(z`−1)|

| supp(z`−1)∪supp(z`−1)| . We allowfor a small overlap of the parts, o(z`−1, z`−1) < 0.2.

The histograms are updated for all inference graphs GIand all the admissible pairs of hypotheses at layer Z`−1.

The choice of r`. The correlation between the relativepositions of the hypotheses is the highest at relativelysmall distances as depicted in Fig. 7. Depending on thefactors of the spatial contraction, the radii r` in this paperare set to 8 for layer 2, 12 for the higher layers, and 15 forthe final, object layer. Note that the circular regions aredefined at spatial resolution of layer `− 1 which meansthat the area, if projected back to the original resolution,covers a much larger area in an image.

5.2.2 Learning compositions of parts

From the learned pair-wise spatial distributions we formduplets via the distribution modes. Fig. 6 illustrates theclustering procedure. For each mode we estimate themean µ` and variance Σ` by fitting a Gaussian distri-bution around it. Each of these modes thus results ina two-part composition with initial parameters: P = 2,θapp

`1(ωì ) = 1, θapp`2(ω`j) = 1 and θg

`2 = (µ`,Σ`) (the first,

reference part, in a composition is always assigned thedefault parameters as explained in Sec. 3). For the higherlayers, ` > 3, where the co-occurrences are sparser,we additionally smooth the learned histograms prior toclustering, to regularize the data.

We can now perform inference with the complete setof duplets. Define a duplet co-occurrence for a particulartraining image neighborhood Ik as a set of duplets thatbest explain Ik (the likelihood of Ik defined by (11) isthe highest for this particular set of duplets) and formsa tree. A duplet co-occurrence forms a composition andwe will denote it by ω`. During training, a list Ω`∗ ofall possible co-occurrences of duplets is kept. Note thatfor each composition in Ω`∗ we also have the (rough)parameters of the distribution (previous paragraph), Θ`

∗.Each ω` is additionally represented by a “count” valuef(ω`) which is taken to be the value of the log-likelihood.Thus, in training, for each neighborhood Ik the count ofthe corresponding ω` is updated accordingly, f(ω`) =f(ω`) + log q(Ik, ω`).

Among Ω`∗ we need to select a set of compositions (avocabulary) yielding the highest score in (17). We adopta greedy approach and further refine it by the MCMCalgorithm. Let Ω`g denote the current set of compositionsselected by the greedy algorithm. At each step of the iter-ation, we do the following: for each neighborhood Ik weupdate the count for the composition ω` ∈ Ω`∗\Ω`g that ismost likely to have generated the neighborhood (has thehighest likelihood), by the difference in its likelihood andthe likelihood under the compositions from ω` ∈ Ω`∗\Ω`g :f(ω`) = f(ω`) + log q(Ik, ω`) − maxω`

i∈Ω`

glog q(Ik, ωì ).

After accounting for all the neighborhoods, we select thecomposition with the highest count.


Fig. 5. Approach to learning the structure and the parameters of the vocabulary at a particular layer.

ω1i = —ω1j = —

ω1i = —ω1j = —

ω1i = —ω1j = —

ω1i = —ω1j = —

ω1i = —ω1j = —

ω1i = —ω1j = —

Fig. 6. Top: Examples of learned histograms h`=2ij . Bottom:

Examples of duplets defined by modes in the spatial histograms.

Fig. 7. Examples of learned histograms h`=2ij for very large

radius, r` = 50. The circle shows the chosen radius, r` = 8.

We next employ a stochastic MCMC algorithm to getthe final vocabulary Ω`. The first state of the Markovchain is the vocabulary Ω`g obtained with the greedyselection. Let Ω`t denote the vocabulary at the currentstate of the chain. At each step, we run the EM toalso get the parameters Θ`

t , where Θ`∗ is always used

to initialize EM. We either exchange/add/remove onecomposition from Ω`t with another one from Ω`∗ \ Ω`t toget the vocabulary Ω`t+1. The vocabulary Ω`t+1 is acceptedas the next state of the Markov chain with probability

min(1, βL(D;Ω`t+1,Θ

`t+1)−L(D;Ω`

t,Θ`t)), β > 1 (20)

according to the Metropolis-Hastings algorithm.The vocabulary at layer ` is finally defined as the Ω`

with maximal value L(D; Ω`,Θ`), after running severaliterations of the M-H algorithm. We usually perform 100iterations.

To train the top, object-layer of compositions, we makeuse of the available supervision, similarly as in theconstellation model [53]. During the greedy selectionof the compositions, we always select the one with the

highest detection accuracy (we use the F-measure).

5.3 Learning a multi-class vocabularyWe learn multiple object classes with the followingstrategy. Learning of the lower layers which are moregeneric is performed over the images of all classes (thusmaximizing sharing between them). Moreover, we preferto learn the lower layers from a set of natural images,which avoids knowing the classes in advance and resultsin a vocabulary which is more in tune to the statistics ofnatural images. Because the general, class-independentstatistics of shapes becomes very sparse in the higherlayers, we learn layers 4 and higher sequentially — byupdating the vocabulary with the compositions learnedfor one class after another. When learning each class weuse the vocabulary learned so far and only add thosecompositions to each of the layers which are still neededto represent the new object class.

When training the higher layers we use the boundingbox information of the objects. We additionally scalethe images so that the diagonal of the bounding box isapproximately 250 pixels. The size of the training imageshas a direct influence on the number of layers beinglearned (due to compositionality, the number of layersneeded to represent the whole shapes of the objects islogarithmic in the number of extracted edge features inthe image). In order to learn a 6-layer hierarchy, the 250pixel diagonal constraint has proven to be a good choice.

Once we have learned the multi-class vocabulary, wecould, in principle, run the EM algorithm to re-learnthe parameters over the complete hierarchy in a similarfashion as in the Hierarchical HMMs [50]. However, wehave not done so in this paper.

5.3.1 Learning the thresholds for the testsGiven that the object class representation is known, wecan learn the thresholds τω` to be used in our (approxi-mate) inference algorithm (Sec. 4.2.3).

We use a similar idea to that of Amit et al. [39] andFleuret and Geman [37]. The main goal is to learn thethresholds in way that nothing is lost with respect to theaccuracy of detection, while at the same time optimizingfor the efficiency of inference.

Specifically, by running the inference algorithm onthe set of class training images Ik, we obtain the


object class likelihoods q(Ik, ck), the correspondingmost probable hidden trees of states T (Ik, ck) andtheir probabilities δ(Ik, ck). For each composition ω`

in the vocabulary, we then define τω` as the minimalprobability obtained in the training set:

τω` = mink

minz`=(ω`,x`)∈T (Ik,ck)

δ(Ik(x`), ω`) (21)

For a better generalization, however, one can rather takea certain percentage of the value on the right [37].

6 EXPERIMENTAL RESULTS

The evaluation is separated into two parts. We first showthe capability of our method to learn generic shapestructures from natural images and apply them to aobject classification task. Second, we utilize the approachfor multi-class object detection on 15 object classes.

All experiments were performed on one core on aIntel Xeon-4 CPU 2.66 Ghz computer. The method isimplemented in C++. Filtering is performed on CUDA.

6.1 Natural statistics and object classification

We applied our learning approach to a collection of1500 natural images. Learning was performed only upto layer 3, with the complete learning process takingroughly 5 hours. The learned hierarchy consisted of 160compositions on Layer 2 and 553 compositions on Layer3. A few examples from both layers are depicted inFig. 9 (note that the images shown are only the meanshapes generated by the generative models). The learnedfeatures include corners, end-stopped lines, various cur-vatures, T- and L-junctions. Interestingly, these structuresresemble the ones predicted by the Gestalt rules [20].

To put the proposed hierarchical framework in re-lation to other categorization approaches which focusprimarily shape, the learned 3−layer vocabulary wastested on the Caltech-101 database [54]. The Caltech-101dataset contains images of 101 different object categorieswith the additional background category. The numberof images varies from 31 to 800 per category, with theaverage image size of roughly 300× 300 pixels.

Each image was processed at 3 scales spaced apart by√2. The likelihoods of the hidden states were combined

with a linear one-versus-all SVM classifier [55]. This wasdone as follows: for each image a vector of a dimensionequal to the number of compositions at the particularlayer was formed. Each dimension of the vector, whichcorresponds to a particular type of a composition (e.g. anL-junction) was obtained by summing over all the likeli-hoods of the hidden states coding this particular type ofcomposition. To increase the discriminative information,a radial sampling was used similarly as in [56]: eachimage was split into 5 orientation bins and 2 distancesfrom the image center, for which separate vectors (asdescribed previously) were formed and consequentlystacked together into one, high-dimensional vector.

The results, averaged over 8 random splits of trainand test images, are reported in Table 1 with classifica-tion rates of existing hierarchical approaches shown forcomparison. For 15 and 30 training images we obtained a60.5% and 66.5% accuracy, respectively, which is slightlybetter than the most related hierarchical approaches [11],[57], [29], comparable to those using more sophisticateddiscriminative methods [58], [59], and slightly worsethan those using additional information such as coloror regions [60].

We further tested the classification performance byvarying the number of training examples. For testing,50 examples were used for the classes where this waspossible and less otherwise. The classification rate wasnormalized accordingly. In all cases, the result was av-eraged over 8 random splits of the data. The results arepresented and compared with existing methods in Fig. 8.We only compare to approaches that use shape informa-tion alone and not also color and texture. Overall, betterperformance has been obtained in [61].

TABLE 1Average classification rate (in %) on Caltech 101.

Ntrain = 15 Ntrain = 30

Mutch et al. [57] 51 56Ommer et al. [11] / 61.3Ahmed at al. [59] 58.1 67.2Yu at al. [58] 59.2 67.4Lee at al. [31] 57.7 65.4Jarrett at al. [29] / 65.5our approach 60.5 66.5

0 5 10 15 20 25 3010

20

30

40

50

60

70

Ntrain

Per

form

ance

(%

)

Caltech−101 Performance

ApproachesGriffin, Holub & Perona (2007)Zhang, Berg, Maire & Malik (2006)Lazebnik, Schmid & Ponce (2006)Wang, Zhang, Fei−Fei (2006)Grauman & Darrell (2005)Mutch and Lowe (2006) Our approach

Fig. 8. Classification performance on Caltech-101.

6.2 Object class detectionThe approach was tested on 15 diverse object classesfrom standard recognition datasets. The basic informa-tion is given in Table 2. These datasets are known tobe challenging because they contain a high amount ofclutter, multiple objects per image, large scale differencesof objects and exhibit a significant intra class variability.

For training we used the bounding box informationof objects or the masks if they were available. Each


object was resized so that its diagonal in an image wasapproximately 250 pixels. For testing, we up-scaled eachtest image by a factor 3 and used 6 scales for objectdetection. When evaluating the detection performance, adetection is counted as correct, if the predicted boundingbox bfg coincides with the ground truth bgt more than50%: area(bfg∩bgt)

area(bfg∪bgt) > 0.5. On the ETH dataset and INRIAhorses this threshold is lowered to 0.3 to enable a faircomparison with the related work [44]. The performanceis given either with recall at equal error rate (EER) orpositive detection rate at low FPPI, depending on thetype of results reported on these datasets thus-far.

6.2.1 Single class learning and performanceWe first evaluated our approach to learn and detecteach object class individually. We report the training andinference times, and the accuracy of detection, which willthen be compared to the multi-class case in Subsec. 6.2.2to test the scalability of the approach.

Training time. To train a class it takes on average 20−25minutes. For example, it takes 23 minutes to train onApple logo, 31 for giraffe, 17 for swan, 25 for cow, and 35for the horse class. For comparison, in Shotton et al. [4],training on 50 horses takes roughly 2 hours (on a 2.2GHz machine using a C# implementation).

Inference time. Detection for each individual class takesfrom 2− 4 seconds per image, depending on the size ofthe image and the amount of texture it contains. Otherrelated approaches report approx. 5 to 10 times highertimes (in seconds): [35]: 20− 30, [38]: 16.9, [62]: 20, [1]:12− 18, however, at slightly older hardware.

Detection performance. The ETH experiments are per-formed in a 5-fold cross-validation obtained by sampling5 subsets of half of the class images at random. The testset for evaluating detection consists of all the remain-ing images in the dataset. The detection performanceis given as the detection rate at the rate of 0.4 false-positives per image (FPPI), averaged over the 5 trials asin [44]. The detection performance is reported in Table 3.Similarly, the results for the INRIA horses are given in a5-fold cross-validation obtained by sampling 5 subsets of50 class images at random and using the remaining 120for testing. The test set also includes 170 negative imagesto allow for a higher FPPI rate. With respect to [44], weachieve a better performance for all classes, most notablyfor giraffes (24.7%). Our method performs comparablyto a discriminative framework by Fritz and Schiele [62].Better performances have recently been obtained by Majiand Malik [63] (93.2%) and Ommer and Malik [52](88.8%) using Hough voting and SVM-based verification.To the best of our knowledge, no other hierarchicalapproach has been tested on this dataset so far.

For the experiments on the rest of the datasets wereport the recall at EER. The results are given in Table 3.Our approach achieves competitive detection rates withrespect to the state-of-the-art. Note also that [34], [6] used150 training examples of motorbikes, while we only used50 (to enable a fair comparison with [3] on GRAZ).

While the performance in a single-class case is compa-rable to the current state-of-the-art, the main advantageof our approach are its computational properties whenthe number of classes is higher, as demonstrated next.

6.2.2 Multi-class learning and performanceTo evaluate the scaling behavior of our approach wehave incrementally learned 15 classes one after another.

The learned vocabulary. A few examples of the learnedshapes at layers 4 to 6 are shown in Fig.10 (only samplesfrom the generative model are depicted). An exampleof a complete object-layer composition is depicted inFig. 14. It can be seen that the approach has learnedthe essential structure of the class well, not missing anyimportant shape information.

Degree of composition sharing among classes. To see howshared are the vocabulary compositions between classesof different degrees of similarity, we depict the learnedvocabularies for two visually similar classes (motor-bike and bicycle), two semi-similar classes (giraffe andhorse), and two dissimilar classes (swan and car front)in Fig. 11. The nodes correspond to compositions and thelinks denote the compositional relations between themand their parts. The green nodes represent the compo-sitions used by both classes, while the specific colorsdenote class specific compositions. If a spatial relationis shared as well, the edge is also colored green. Thecomputational advantage of our hierarchical representa-tion over flat ones [3], [6], [2] is that the compositionsare shared at several layers of the vocabulary, whichsignificantly reduces the overall complexity of inference.Even for the visually dissimilar classes, which may nothave any complex parts in common, our representationis highly shared at the lower layers of the hierarchy.

To evaluate the shareability of the learned composi-tions among the classes, we use the following measure:

deg share(`) =1|Ω`|

∑ω`∈Ω`

(# of classes that use ω`)− 1# of all classes− 1

,

defined for each layer ` separately. By “ω` used by classc” it is meant that the probability of ω` under c is notequal to zero. To give some intuition behind the measure:deg share = 0 if no composition from layer ` is shared(each class uses its own set of compositions), and it is 1if each composition is used by all the classes. Fig. 13 (b)plots the values for the learned 15−class representation.Beside the mean (which defines deg share), the plotsalso show the standard deviation.

The size of the vocabulary. It is important to test howthe size of the vocabulary (the number of compositionsat each layer) scales with the number of classes, sincethis has a direct influence on inference time. We reportthe size as a function of the number of learned classesin Fig. 12. For the “worst case” we take the independenttraining approach: a vocabulary for each class is learnedindependently of other classes and we sum over thesizes of these separate vocabularies. One can observe


a logarithmic tendency especially at the lower layers.This is particularly important because the complexityof inference is much higher for these layers (becausethey contain less discriminative compositions which aredetected more numerously in images). Although the fifthlayer contains highly class specific compositions, one canstill observe a logarithmic increase in size. The final,object layer, is naturally linear in the number of classes,but it does, however, learn less object models than is thenumber of all training examples of objects in each class.

We further compare the scaling tendency of our ap-proach with the one reported for a non-hierarchical rep-resentation by Opelt et al. [3]. The comparison is givenin Fig. 13 (a) where the worst case is taken as in [3]. Wecompare the size of the class-specific vocabulary at layer5, where the learned compositions are of approximatelythe same granularity as the features used in [3]. We ad-ditionally compare the overall size of the vocabulary (asum of the sizes over all layers). Our approach achievesa substantially better scaling tendency, which is due tosharing at multiple layers of representation. This, on theone hand, compresses the overall representation, whileit also attains a higher variability of the compositions inthe higher layers and consequently, a lower number ofthem are needed to represent the classes.

Fig. 13 (d) shows the storage demands as a functionof the number of learned classes. This is the actual sizeof the vocabulary stored on a hard disk. Notice that the15-class hierarchy takes only 1.6Mb on disk.

Inference time. We further test how the complexity ofinference increases with each additional class learned.We randomly sample ten images per class and report thedetection times averaged over all selected images. Thetimes are reported as a function of the number of learnedclasses. The results are plotted in Fig. 13 (c), showing thatthe running times are significantly faster than the “worstcase” (the case of independent class representation, inwhich each separate class vocabulary is used to detectobjects). It is worth noting, that it takes only 16 secondsper image to apply a vocabulary of all 15 classes.

Detection performance. We additionally test the multi-class detection performance. The results are presented inTable 4 and compared with the performance of the singleclass vocabularies. The evaluation was the following. Forthe single class case, a separate vocabulary was trainedon each class and evaluated for detection independentlyof other classes. For the multi-class case, a joint multi-class vocabulary was learned (as explained in Sec. 6.2.2)and detection was performed as proposed in Sec. 5.3,that is, we allowed for competition among hypothesesexplaining partly the same regions in an image.

Overall, the detection rates are slightly worse in themulti-class case, although in some cases the multi-classcase outperformed the independent case. The main rea-son for a slightly reduced performance is due to the factthat the representation is generative and is not trained todiscriminate between the classes. Consequently, it doesnot separate similar classes sufficiently well and a wrong

hypothesis may end up inhibiting the correct one. Aspart of the future work, we plan to also incorporate morediscriminative information into the representation andincrease the recognition accuracy in the multi-class case.On the other hand, the increased performance for themulti-class case for some objects is due to feature sharingamong the classes, which results in better regularizationand generalization of the learned representation.

7 SUMMARY AND CONCLUSIONS

We proposed a novel approach which learns a hierarchi-cal compositional shape vocabulary to represent multi-ple object classes in an unsupervised manner. Learningis performed bottom-up, from small oriented contourfragments to whole-object class shapes. The vocabularyis learned recursively, where the compositions at eachlayer are combined via spatial relations to form largerand more complex shape compositions.

Experimental evaluation was two-fold: one that showsthe capability of the method to learn generic shapestructures from natural images and uses them for objectclassification, and another one that utilizes the approachin multi-class object detection. We have demonstrateda competitive classification and detection performance,fast inference even for a single class, and most impor-tantly, a logarithmic growth in the size of the vocabulary(at least in the lower layers) and, consequently, scalabil-ity of inference complexity as the number of modeledclasses grows. The observed scaling tendency of ourhierarchical framework goes well beyond that of a flatapproach [3]. This provides an important showcase thathighlights learned hierarchical compositional vocabular-ies as a suitable form of representing a higher numberof object classes.

8 FUTURE WORK

There are numerous directions for future work. One im-portant aspect would be to include multiple modalitiesin the representation. Since many object classes havedistinctive textures and color, adding this informationto our model would increase the range of classes thatthe method could be applied to. Additionally, modelingtexture could also boost the performance of our currentmodel: since textured regions in an image usually havea lot of noisy feature detections, they are currently moresusceptible to false positive object detections. Having amodel of texture could be used to remove such regionsfrom the current inference algorithm.

Part of our ongoing work is to make the approach scal-able to a large number of object classes. To achieve this,several improvements and extensions are still needed.We will need to incorporate the discriminative infor-mation into the model (to distinguish better betweensimilar classes), make use of contextual information toimprove performance in the case of ambiguous informa-tion (small objects, large occlusion, etc) and use attentionmechanisms to speed-up detection in large complex


Layer 1

Layer 2 Layer 3

Fig. 9. Examples of learned vocabulary compositions (with the exception of a fixed Layer 1) learned on 1500 naturalimages. Only the mean of the compositions are depicted.

Layer 4 Layer 5

Layer 6 (object layer O)

Apple logo person bottle mug swan cup cow horse car bicycle face giraffe

Fig. 10. Examples of samples from compositions at layers 4, 5, and top, object layer learned on 15 classes.

TABLE 2Information on the datasets used to evaluate the detection performance.

dataset ETH shape UIUC Weizmann INRIA TUDclass Apple logo bottle giraffe mug swan car side (multiscale) horse (multiscale) horse motorbike

# train im. 19 21 30 24 15 40 20 50 50# test im. 21 27 57 24 17 108 228 290 115

dataset GRAZclass bicycle side bottle car front car rear cow cup face horse side motorbike mug person

# train im. 45 / 20 50 20 16 50 30 50 / 19# test im. 53 64 20 400 65 16 217 96 400 15 19

images. Furthermore, a taxonomic organization of objectclasses could further improve the speed of detection and,

possibly, also the recognition rates of the approach.


TABLE 3Detection results. On the ETH shape and INRIA horses we report the detection-rate (in %) at 0.4 FPPI averaged overfive random splits train/test data. For all the other datasets the results are reported as recall at equal-error-rate (EER).

ETHshape

class [44] [62] our approachapplelogo 83.2 (1.7) 89.9 (4.5) 87.3 (2.6)bottle 83.2 (7.5) 76.8 (6.1) 86.2 (2.8)giraffe 58.6 (14.6) 90.5 (5.4) 83.3 (4.3)mug 83.6 (8.6) 82.7 (5.1) 84.6 (2.3)swan 75.4 (13.4) 84.0( 8.4) 78.2 (5.4)average 76.8 84.8 83.7

INRIA horse 84.8(2.6) / 85.1(2.2)

class related work our approachUIUC car side, multiscale 90.6 [57] 93.5 [60] 93.5Weizmann horse multiscale 89.0 [4] 93.0 [64] 94.3TUD motorbike 87 [6] 88 [34] 83.2

class [3] [4] our approachface 96.4 97.2 94

GRAZ

bicycle side 72 67.9 68.5bottle 91 90.6 89.1cow 100 98.5 96.9cup 81.2 85 85car front 90 70.6 76.5car rear 97.7 98.2 97.5horse side 91.8 93.7 93.7motorbike 95.6 99.7 93.0mug 93.3 90 90person 52.6 52.4 60.4

(a) (b) (c)

Fig. 11. Sharing of compositions between classes. Each plot shows a vocabulary learned for two object classes. The bottomnodes in each plot represent the 6 edge orientations, one row up denotes the learned second layer compositions, etc. The sixthrow (from bottom to top) are the whole-shape, object-layer compositions (i.e., different aspects of objects in the classes), whilethe top layer is the class layer which pools the corresponding object compositions together. The green nodes denote the sharedcompositions, while specific colors show class-specific compositions. From left to right: Vocabularies for: (a) two visually similarclasses (motorbike and bicycle), (b) two semi-similar object classes (giraffe and horse), (c) two visually dissimilar classes (swanand car front). Notice that even for dissimilar object classes, the compositions from the lower layers are shared between them.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

50

100

150

number of classes

num

ber

of s

hape

mod

els

Size of representation (# classes)

worst case (linear)our approach

Layer 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

50

100

150

200

number of classes

num

ber

of s

hape

mod

els



Layer 3

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

50

100

150

200

250

300

350

number of classes

num

ber

of s

hape

mod

els



Layer 4

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

50

100

150

200

250

300

350

400

number of classes

num

ber

of s

hape

mod

els



Layer 5

Fig. 12. Size of representation (the number of compositions per layer) as a function of the number of learned classes. “Worstcase” denotes the sum over the sizes of single-class vocabularies.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

200

400

600

800

1000

1200

1400

number of classes

num

ber

of v

ocab

ular

y fe

atur

es

Size of representation, flat vs hier.

worst case (linear)Opelt et al. 2008layer 5whole hierarchy

1 2 3 4 5

00.10.20.30.40.50.60.70.80.9

1

Degree of sharing

layer

degr

ee o

f sha

ring

15 classesmotorbike−bikegiraffe−horseswan−car front

2 4 6 8 10 12 140

10

20

30

40

50

60

70

number of classes

infe

renc

e tim

e pe

r im

age

(sec

)

Inference time per image (# classes)


1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

500

1000

1500

2000

2500

3000

number of classes

cum

ulat

ive

size

on

disk

(K

b)

Storage (size on disk)


(a) (b) (c) (d)

Fig. 13. From left to right: (a) A comparison in scaling to multiple classes of the approach by Opelt et al. [3] (flat) and ourhierarchical approach; (b) Degree of sharing for the multi-class vocabulary; (c) Average inference time per image as a function ofthe number of learned classes; (d) Storage (size of the hierarchy stored on disk) as a function of the number of learned classes.


TABLE 4Comparison in detection accuracy for single- and multi-class object representations and detection procedures.

class Apple bottle giraffe mug swan horse cow mbike bicycle car front car side car rear face person cupmeasure detection rate (in %) at 0.4 FPPI recall (in %) at EER

single 88.6 85.5 83.5 84.9 75.8 84.5 96.9 83.2 68.5 76.5 97.5 93.5 94.0 60.4 85.0multi 84.1 80.0 80.2 80.3 72.7 82.3 95.4 83.2 64.8 82.4 93.8 92.0 93.0 62.5 85.0

(a) (b) (c)

Fig. 14. One learned object-layer composition ωO for (a) bottle, (b) giraffe, and (c) bicycle, with the complete model structureshown. The blue patches denote the limiting circular regions, the color regions inside them depict the spatial relations (Gaussians).The nodes are the compositions in the vocabulary. Note that each composition models the distribution over the compositions fromthe previous layer for the appearance of its parts. For clarity, only the most likely composition for the part is decomposed further.

Fig. 15. Examples of detections. The links show the most probable hidden state activation for a particular (top-layer) classdetection. The links are color-coded to denote different classes.

ACKNOWLEDGMENTS

This research has been supported in part by the follow-ing funds: Research program Computer Vision P2-0214(Slovenian Ministry of Higher Education, Science andTechnology) and EU FP7-215843 project POETICON.

REFERENCES

[1] R. Fergus, P. Perona, and A. Zisserman, “Weakly supervised scale-invariant learning of models for visual recognition,” InternationalJ. of Computer Vision, vol. 71, no. 3, pp. 273–303, March 2007.

[2] A. Torralba, K. P. Murphy, and W. T. Freeman, “Sharing visualfeatures for multiclass and multiview object detection,” IEEETrans. Patt. Anal. and Mach. Intell., vol. 29, no. 5, pp. 854–869, 2007.

[3] A. Opelt, A. Pinz, and A. Zisserman, “Learning an alphabet ofshape and appearance for multi-class object detection,” Interna-tional J. of Computer Vision, vol. 80, no. 1, pp. 16–44, 2008.

[4] J. Shotton, A. Blake, and R. Cipolla, “Multi-scale categorical objectrecognition using contour fragments,” IEEE Trans. Patt. Anal. andMach. Intell., vol. 30, no. 7, pp. 1270–1281, 2008.

[5] V. Ferrari, L. Fevrier, F. Jurie, and C. Schmid, “Groups of adjacentcontour segments for object detection,” IEEE Trans. Patt. Anal. andMach. Intell., vol. 30, no. 1, pp. 36–51, 2008.

[6] B. Leibe, A. Leonardis, and B. Schiele, “Robust object detectionwith interleaved categorization and segmentation,” InternationalJ. of Computer Vision, vol. 77, no. 1-3, pp. 259–289, 2008.


[7] G. Bouchard and B. Triggs, “Hierarchical part-based visual objectcategorization,” in CVPR, 2005, pp. 710–715.

[8] S. Ullman and B. Epshtein, Visual Classification by a Hierarchy of Ex-tended Features, ser. Towards Category-Level Object Recognition.Springer-Verlag, 2006.

[9] S. Zhu and D. Mumford, “A stochastic grammar of images.”Foundations and Trends in Computer Graphics and Vision, vol. 2, no. 4,pp. 259–362, 2006.

[10] S. Fidler and A. Leonardis, “Towards scalable representations ofvisual categories: Learning a hierarchy of parts,” in CVPR, 2007.

[11] B. Ommer and J. M. Buhmann, “Learning the compositionalnature of visual objects,” in CVPR, 2007.

[12] J. K. Tsotsos, “Analyzing vision at the complexity level,” Behavioraland Brain Sciences, vol. 13, no. 3, pp. 423–469, 1990.

[13] Y. Amit, 2d Object Detection and Recognition: Models, Algorithms andNetworks. MIT Press, Cambridge, 2002.

[14] M. Leordeanu, R. Sukhtankar, and M. Hebert, “Object and cat-egory recognition from pairwise relations between features,” inCVPR, 2007.

[15] A. Shokoufandeh, L. Bretzner, D. Macrini, M. Demirci, C. Jonsson,and S. Dickinson, “The representation and matching of categoricalshape,” Computer Vision and Image Understanding, vol. 103, pp.139–154, 2006.

[16] S. Sarkar and K. Boyer, Computing Perceptual Organization inComputer Vision. Singapore: World Scientific, 1994.

[17] S. Geman, D. Potter, and Z. Chi, “Composition systems,” Quarterlyof Applied Mathematics, vol. 60, no. 4, pp. 707–736, 2002.

[18] J. Schwartz and P. Felzenszwalb, “Hierarchical matching of de-formable shapes., booktitle = cvpr, year = 2007.”

[19] S. Edelman, N. Intrator, and J. S. Jacobson, “Unsupervised learn-ing of visual structure,” in Biologically Motivated Computer Vision,2002, pp. 629–642.

[20] M. Wertheimer, “Untersuchungen zur lehre von der gestalt.”Psychologische Forschung: Zeitschrift fuer Psychologie und ihre Gren-zwissenschaften, vol. 4, no. 1, pp. 301–350, 1923.

[21] G. J. Ettinger, “Hierarchical object recognition using libraries ofparameterized model sub-parts,” MIT, Tech. Rep., 1987.

[22] S. Dickinson, “The evolution of object categorization and the chal-lenge of image abstraction,” in Object Categorization: Computer andHuman Vision Perspectives, S. Dickinson, A. Leonardis, B. Schiele,and M. J. Tarr, Eds. Cambridge University Press, 2009.

[23] J. Utans, “Learning in compositional hierarchies: Inducing thestructure of objects from data,” in NIPS, 1994, pp. 285–292.

[24] K. Fukushima, S. Miyake, and T. Ito, “Neocognitron: a neuralnetwork model for a mechanism of visual pattern recognition,”IEEE Systems, Man and Cyber., vol. 13, no. 3, pp. 826–834, 1983.

[25] M. Riesenhuber and T. Poggio, “Hierarchical models of objectrecognition in cortex,” Nature Neuroscience, vol. 2, no. 11, pp. 1019–1025, 1999.

[26] T. Serre, L. Wolf, S. Bileschi, M. Riesenhuber, and T. Poggio,“Object recognition with cortex-like mechanisms,” IEEE Trans.Patt. Anal. and Mach. Intell., vol. 29, no. 3, pp. 411–426, 2007.

[27] T. Masquelier and S. J. Thorpe, “Unsupervised learning of visualfeatures through spike timing dependent plasticity,” PLoS Com-putational Biology, vol. 3, no. 2, pp. 832–838, 2007.

[28] M. A. Ranzato, F. J. Huang, Y. L. Boureau, and Y. LeCun, “Un-supervised learning of invariant feature hierarchies with applica-tions to object recognition,” in CVPR, 2007.

[29] K. Jarrett, K. Kavukcuoglu, M. A. Ranzato, and Y. LeCun, “What isthe best multi-stage architecture for object recognition?” in ICCV,2009.

[30] G. E. Hinton, “Learning multiple layers of representation,” Trendsin Cognitive Sciences, vol. 11, no. 10, pp. 428–434, 2007.

[31] R. R. Honglak Lee, Roger Grosse and A. Y. Ng, “Convolutionaldeep belief networks for scalable unsupervised learning of hier-archical representations,” in ICML, 2009, pp. 609–616.

[32] E. Sudderth, A. Torralba, W. T. Freeman, and A. Willsky, “De-scribing visual scenes using transformed objects and parts,” In-ternational J. of Computer Vision, vol. 77, no. 1-3, pp. 291–330, 2008.

[33] B. Epshtein and S. Ullman, “Semantic hierarchies for recognizingobjects and parts,” in CVPR, 2007.

[34] K. Mikolajczyk, B. Leibe, and B. Schiele, “Multiple object classdetection with a generative model,” in CVPR, 2006, pp. 26–36.

[35] S. Todorovic and N. Ahuja, “Unsupervised category modeling,recognition, and segmentation in images,” IEEE Trans. Patt. Anal.and Mach. Intell., (in press).

[36] F. Scalzo and J. H. Piater, “Statistical learning of visual featurehierarchies,” in Workshop on Learning, CVPR, 2005.

[37] F. Fleuret and D. Geman, “Coarse-to-fine face detection,” Interna-tional J. of Computer Vision, vol. 41, no. 1/2, pp. 85–107, 2001.

[38] L. Zhu, C. Lin, H. Huang, Y. Chen, and A. Yuille, “Unsupervisedstructure learning: Hierarchical recursive composition, suspiciouscoincidence and competitive exclusion,” in ECCV, vol. 2, 2008,pp. 759–773.

[39] Y. Amit, D. Geman, and X. Fan, “A coarse-to-fine strategy formulticlass shape detection,” IEEE Trans. Patt. Anal. and Mach.Intell., vol. 26, no. 12, pp. 1606–1621, 2004.

[40] Y. Amit and D. Geman, “A computational model for visualselection,” Neural Computation, vol. 11, no. 7, pp. 1691–1715, 1999.

[41] M. Marszałek and C. Schmid, “Constructing category hierarchiesfor visual recognition,” in ECCV, ser. LNCS, vol. IV. Springer,2008, pp. 479–491.

[42] P. Zehnder, E. K. Meier, and L. V. Gool, “An efficient shared multi-class detection cascade,” in BMVC, 2008.

[43] R. C. Nelson and A. Selinger, “Large-scale tests of a keyed,appearance-based 3-d object recognition system,” Vision Research,vol. 38, no. 15-16, pp. 2469–2488, 1998.

[44] V. Ferrari, L. Fevrier, F. Jurie, and C. Schmid, “Accurate objectdetection with deformable shape models learnt from images,” inCVPR, 2007.

[45] R. Fergus, P. Perona, and A. Zisserman, “A sparse object categorymodel for efficient learning and exhaustive recognition,” in CVPR,vol. 1, 2005, pp. 380–397.

[46] S. Fidler, G. Berginc, and A. Leonardis, “Hierarchical statisticallearning of generic parts of object structure,” in CVPR, 2006, pp.182–189.

[47] S. Fidler, M. Boben, and A. Leonardis, “Similarity-based cross-layered hierarchical representation for object categorization,” inCVPR, 2008.

[48] ——, “Learning hierarchical compositional representations of ob-ject structure,” in Object Categorization: Computer and Human VisionPerspectives, S. Dickinson, A. Leonardis, B. Schiele, and M. J. Tarr,Eds. Cambridge University Press, 2009.

[49] N. Petkov, “Biologically motivated computationally intensive ap-proaches to image pattern recognition,” Future Generation Com-puter Systems, vol. 11, no. 4-5, pp. 451–465, 1995.

[50] S. Fine, Y. Singer, and N. Tishby, “The hierarchical hidden markovmodel: Analysis and applications,” Machine Learning, vol. 32,no. 1, pp. 41–62, 1998.

[51] G. Blanchard and D. Geman, “Hierarchical testing designs forpattern recognition,” Annals of Statistics, vol. 33, no. 3, pp. 1155–1202, 2005.

[52] B. Ommer and J. Malik, “Multi-scale object detection by clusteringlines,” in ICCV, 2009.

[53] M. Weber, M. Welling, and P. Perona, “Towards automatic discov-ery of object categories,” in CVPR, 2000, pp. 2101–2108.

[54] L. Fei-Fei, R. Fergus, and P. Perona, “Learning generative visualmodels from few training examples: an incremental bayesian ap-proach tested on 101 object categories.” in IEEE CVPR, Workshopon Generative-Model Based Vision, 2004.

[55] T. Joachims, “Making large-scale support vector machine learningpractical,” pp. 169–184, 1999.

[56] H. Zhang, A. C. Berg, M. Maire, and J. Malik, “Svm-knn: Dis-criminative nearest neighbor classification for visual categoryrecognition,” in CVPR, vol. 2, 2006, pp. 2126–2136.

[57] J. Mutch and D. G. Lowe, “Multiclass object recognition withsparse, localized features,” in CVPR, 2006, pp. 11–18.

[58] K. Yu, W. Xu, and Y. Gong, “Deep learning with kernel regular-ization for visual recognition,” in NIPS, 2008, pp. 1889–1896.

[59] A. Ahmed, K. Yu, W. Xu, Y. Gong, and E. P. Xing, “Traininghierarchical feed-forward visual recognition models using transferlearning from pseudo tasks,” in ECCV, vol. III, 2009, pp. 69–82.

[60] N. Ahuja and S. Todorovic, “Connected segmentation tree – ajoint representation of region layout and hierarchy,” CVPR, 2008.

[61] M. Varma and D. Ray, “Learning the discriminative power-invariance trade-off,” in ICCV, 2007.

[62] M. Fritz and B. Schiele, “Decomposition, discovery and detectionof visual categories using topic models,” in CVPR, 2008.

[63] S. Maji and J. Malik, “Object detection using a max-margin houghtransform,” in CVPR, 2009.

[64] J. Shotton, A. Blake, and R. Cipolla, “Efficiently combining con-tour and texture cues for object recognition,” in BMVC, 2008.

SUBMITTED TO IEEE TRANSACTIONS ON PATTERN …fergus/drafts/pamipaper.pdfapproach learns hierarchical...

Documents

Transcript of SUBMITTED TO IEEE TRANSACTIONS ON PATTERN …fergus/drafts/pamipaper.pdfapproach learns hierarchical...