Local Image Representations Using Pruned Salient Pointswith Applications to CBIR

Hui Zhang, Rouhollah Rahmani, Sharath R. Cholleti, Sally A. GoldmanDepartment of Computer Science and Engineering,Washington University, St. Louis, MO 63130 USA

{huizhang, rahmani, cholleti, sg}@wustl.edu

ABSTRACTSalient points are locations in an image where there is asignificant variation with respect to a chosen image feature.Since the set of salient points in an image capture impor-tant local characteristics of that image, they can form thebasis of a good image representation for content-based imageretrieval (CBIR). The features for a salient point should rep-resent the local characteristic of that point so that the sim-ilarity between features indicates the similarity between thesalient points. Traditional uses of salient points for CBIRassign features to a salient point based on the image fea-tures of all pixels in a window around that point. However,since salient points are often on the boundary of objects,the features assigned to a salient point often involve pixelsfrom di!erent objects. In this paper, we propose a CBIRsystem that uses a novel salient point method that both re-duces the number of salient points using a segmentation asa filter, and also improves the representation so that it is amore faithful representation of a single object (or portion ofan object) that includes information about its surroundings.We also introduce an improved Expectation Maximization-Diverse Density (EM-DD) based multiple-instance learningalgorithm. Experimental results show that our CBIR tech-niques improve retrieval performance by !5%-11% as com-pared with current methods.

Categories and Subject Descriptors: H.3.1, H.3.3 [In-formation Systems]: Information Storage and Retrieval [Con-tent Analysis and Indexing, Information Search and Re-trieval]

General Terms: Algorithms, Experimentation.

Keywords: Feature Representation, Salient Points, Inter-est Points, Image Retrieval, Content-Based Image Retrieval,Multiple Instance Learning

1. INTRODUCTIONIn Content-Based Image Retrieval (CBIR) applications,

the user typically provides an image (or a set of images)

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.MM’06, October 23–27, 2006, Santa Barbara, California, USA.Copyright 2006 ACM 1-59593-447-2/06/0010 ...$5.00.

Figure 1: A sample query image.

with no indication of which portion of the image is of inter-est. Thus a search in classic CBIR often relies upon a globalview of the image. Localized CBIR [26] has been definedas a task where the user is only interested in a portion ofthe image and the rest is irrelevant. For example, a userwho gives the image in Figure 1 as a query image, mightbe interested in the large black rock, the brick walkway, thebuilding, the grass, the painted pot, or the pink granite poston which the pot has been placed. Unless the user explic-itly marks the region of interest or the region of interest isat a fixed location, localized CBIR systems must use rele-vance feedback (or some mechanism) to obtain a small set ofpositively and negatively labeled images to learn from. Thefocus of our work is to design a localized CBIR system thatdoes not require the user to indicate the region of interest.A CBIR system that can obtain good retrieval performancefor localized CBIR tasks without the user explicitly mark-ing the region of interest must use an image representationthat captures local characteristics of the image, and have aranking algorithm that can learn which portion of the imageis of interest to the user.

To capture local characteristics of an image, many CBIRsystems either subdivide the image into fixed blocks [30,22, 23], or more commonly partition the image into dif-ferent meaningful regions by applying a segmentation al-gorithm [32, 26]. In both cases, each region of the image isrepresented as a feature vector of feature values extractedfrom that region. Other CBIR systems extract salient points(also known as interest points) [29, 21, 33, 15, 17, 34], whichare locations in an image where there is a significant vari-ation with respect to a chosen image feature. With salientpoint-based methods, there is one feature vector created foreach salient point. These representations enable a retrievalmethod to have a representation of di!erent local regions ofthe image, and thus these images can be searched based ontheir local characteristics.

Usually the performance of a segmentation-based methoddepends highly on the quality of the segmentation. Espe-cially, a segmentation based representation usually measuresfeatures on a per-segment basis, and the average features of

all pixels in a segment are often used as the features of thatsegment [32, 26]. Therefore, this representation requireshigh quality segmentations because small areas of incorrectsegmentation might make the representation very di!erentfrom that of the real object.

The features used to represent each salient point must besuch that similarity between feature vectors indicates thesimilarity between the salient points. While there are manydi!erent similarity metrics that are used, in this paper wefocus on finding feature representations that lead to goodresults when using the Euclidean similarity measure.

Traditional uses of salient points for CBIR assign featuresto a salient point based on the image features of all pixelsin a window around the salient point [29, 21, 33, 15, 17, 34].However, since salient points are often on the boundary ofobjects, the features assigned to a salient point often involvepixels from di!erent objects. If all portions of the image areimportant for retrieval, then finding a good representationfor individual segments is not necessary, but for localizedCBIR it is important to faithfully represent local regions.Another drawback of using traditional salient points extrac-tion methods is these points often gather at textured portionof the image, or on edges where the change of image featureis significant, so many salient points capture the same por-tion of the image.

While the representation of an image is a very importantcomponent of a CBIR system, the method used to rank theimages is also crucial to the performance. Some CBIR sys-tems define a similarity metric which is used to rank allimages in a repository based on their similarity to a sin-gle query image [32]. However, such methods cannot workwell for localized CBIR since a single image cannot indicatewhich portion is of interest. Relevance feedback can provideadditional images which could be used to determine whatdistinguishes images the user likes from those that are notdesirable. Yet many methods use relevance feedback to trainthe parameters in a similarity metric which is based uponthe entire image [4, 12, 13, 27]. For example, most CBIRmethods that use salient points rank the images based onthe number of matching salient points. Suppose that thequery image of Figure 1 is given by a user interested in find-ing other images that contain the painted pot. Ranking allimages based on the number of matching salient points withthis query image would gives equal importance to all areaswith salient points. A localized CBIR system must learn aranking function that will focus upon the salient points onthe pot.

There are three key contributions of this paper. (1) Weuse image segmentation to form a mask that limits the num-ber of salient points in each segment while maintaining thediversity of the salient points. (2) We use the local charac-teristics of a salient point to determine how to split the win-dow into di!erent sub-windows, and assign each sub-windowfeatures based on both it and its neighboring sub-window.Our experimental results demonstrate that both of the tech-niques we introduce for salient point based representationssignificantly improve retrieval accuracies. (3) We introducean improved EM-DD algorithm to learn the hypothesis usedto rank the images and apply this to both a segmentationbased and salient point based image representation. Theresults presented in this paper for both the segmentation-based and salient-point based representation improve uponthe best known results.

The remainder of this paper is organized as follows. Sec-tion 2 discuss background. Section 3 discusses related workin terms of the framework provided in Section 2. Section 4presents our salient point representation techniques. Theimproved EM-DD learning algorithm for retrieval is dis-cussed in Section 6. Experimental results are presented inSection 7. Finally, Section 8 contains a conclusion and dis-cussion of future work.

2. BACKGROUNDIn this section we describe background material that will

be helpful in describing our work and relating it to priorwork. Appropriate image representation and ranking aretwo critical components for a localized CBIR system. Mul-tiple Instance Learning (MIL) is particularly suitable forlocalized CBIR when the user does not indicate the regionsof interest.

2.1 Multiple-Instance LearningThe multiple-instance learning model [6] was introduced

for problems in which each example is a set (or bag) of fea-ture vectors (or points) and there is ambiguity as to whichfeature vector(s) within the bag are important. MIL is wellsuited for localized CBIR since there is natural ambiguityas to what portion of each image is important to the user.When applied to CBIR, each bag corresponds to an image,and each feature vector in the bag corresponds to a por-tion (region or salient point) of the image. MIL-based ap-proaches have been shown to be valuable for both localizedCBIR tasks and also more classical CBIR tasks using broadimage categories such as natural scenes [22, 23, 38, 1, 3, 26].

2.2 CBIR Image RepresentationsWe distinguish CBIR systems based on whether they rep-

resent an image as a single feature vector or as a bag offeature vectors.

Single Feature Vector methods represent the entire im-age as one feature vector. For example, a color his-togram [4, 9, 19] defined over the entire image is sucha representation. Instead of defining a feature vectorfor the entire image, some work just uses a portion ofthe image that is user-defined or pre-defined. For ex-ample, Kim et al. [16] only consider the object in thecenter of the image.

Multiple Feature Vectors methods represent the imageas a bag of feature vectors. While many methods thatuse multiple feature vectors apply MIL algorithms, asdiscussed in Section 2.3, this is not always the case.The methods using multiple feature vectors to repre-sent the image can be further distinguished based onthe way in which the portion of the image associatedwith each feature vector is determined.

Pre-defined Blocks: Some methods define blocks ofthe image independently of the image content andassociate a feature vector with each block. Theseblocks can define a partition of the image or over-lap each other. Method that use this approachinclude: regions of interest (ROI) and SpatialLayout [30], as well as single-blob with neighbors(SBN) [22, 23].

Region-based: These methods use a segmentationalgorithm to partition the image into regions thatare similar in color, texture, or other features.Methods that use this approach include: SIM-PLIcity [32] and Accio [26].

Salient Points: Salient points are locations in an im-age where there is a significant variation withrespect to a chosen image feature. When usingsalient points only the portions of the image nearthe salient points are represented. Recently therehas been significant work in using salient pointsfor CBIR [29, 21, 33, 15, 17, 34].

2.3 CBIR Ranking MethodsThe second component we use to classify CBIR systems

is how they construct the ranking function. We divide theseinto global and local methods.

Global Ranking Methods use all “meaningful” portionsof the image in the ranking function. Observe thatwhile salient point based methods only use the por-tions of the image around the salient points, if theranking method is based upon the number of salientpoints that match those of the query image, then it isa global method since all portions of the image aroundthe salient points are used in ranking the images. Thesemethods can further be divided into two sub-categories.

Fixed Similarity Measure: A non-adaptive similar-ity measure is defined over the feature vector(s)which is then use to rank the images from themost to least similar with respect to the query im-age. Such a method is typically used when there isjust a single query image (and no relevance feed-back is used). For example, a cosine similaritymeasure can be used between the vectors definedby a color histogram. Some very sophisticatedsimilarity measures have been defined. For exam-ple, the integrated region matching (IRM) algo-rithm [14] used within SIMPLIcity [32] computesa global similarity measure based on a weightedsum of region-to-region distances with the weightsselected to minimize the distance while ensuringthat all regions are matched.

Trained Similarity Measure: A machine learningalgorithm uses labeled images (generally obtainedvia relevance feedback) to improve the perfor-mance of the ranking algorithm by optimizingparameter value(s) of a parameterized similaritymeasure. For example traditional relevance feed-back often re-weights the features when the imageis represented using a single feature vector suchas a color histogram [4, 12, 13, 27]. More recentlysuch approaches have been extended to represen-tations using multiple feature vectors. ROI andSpatial Layout [30] both partition the image intopre-defined blocks and employ standard relevancefeedback at the block level using the spatiallycorresponding blocks of the feedback images. InROI, the user is asked to draw a bounding boxaround the region of interest in the query imagewhich is used to weight each block relative to howmuch of its area is in the bounding box.

Local Ranking Methods: Apply MIL algorithms to la-beled images that are represented by a set of featurevectors. By using such an approach the ranking func-tions can focus on the feature vectors that are rele-vant to the user’s query. For example, a salient pointbased method that learns which subset S of the salientpoints are contained in desirable images and ranks im-ages based on how many of the salient points in S arecontained in an image uses a local ranking method.

The distinction between a localized CBIR system versusstandard CBIR system is based upon whether or not thesimilarity metric used for ranking is global (i.e. combinesproperties from all regions/blocks/salient points of the im-age) or whether it depends on a subset of them. One canobtain a localized CBIR system by either representing theimage using a single feature vector that depends upon auser-defined (or pre-defined) area of focus, or representingthe image as a bag of feature vectors and using a local rank-ing method.

3. RELATED WORKOur goal is to develop a localized CBIR system for which

the user does not need to define an area of focus, and thedesired object is not at any fixed location. Thus, we focusour discussion of previous work on similar CBIR systems.

Maron and Lozano-Perez [22] were the first to apply MILto a CBIR task. They represented the image using a bagof feature vectors based on nine pre-defined, uniform-size,overlapping regions that consist of a single central block andfour neighboring blocks. They also introduce the diversedensity (DD) MIL algorithm. Their approach was applied toa data set of natural scene images (such as a waterfalls) [23].

Yang et al. [36] used a di!erent set of pre-defined blocks torepresent the image and evaluated the quality of a hypoth-esis with a weighted correlation similarity measure insteadof the diverse density measure. They divided the image into20 pre-defined overlapping rectangular pieces (varying fromthe entire image, to 25% of the image). This process wasperformed on the image and its left-right mirror.

Zhang and Goldman [37] introduced the EM-DD algo-rithm which treats the knowledge of which instance corre-sponds to the label of the bag as a missing attribute and ap-plies a variation of the EM algorithm[5] to convert the MILproblem to a standard supervised learning problem by in-troducing as a hidden variable a selector variable indicatingwhich feature vector in the bag is positive. Zhang et al. [38]apply EM-DD to CBIR using a region-based approach torepresent each image as a bag. Specifically, they use a seg-mentation algorithm to partition the image into segmentsand represent each segment as a feature vector containingthe average color and wavelet-based texture features. Theyalso modified EM-DD to rank the images in the test set us-ing the average label from all hypotheses returned from themultiple runs of EM used within EM-DD.

Huang et al. [13] presented a variation of Zhang and Gold-man’s approach that incorporated a di!erent segmentationalgorithm, and a neural network based MIL algorithm. An-drews et al. [1] introduced two new MIL algorithms, MI-SVM and mi-SVM, that also use an EM-based approach.However, they use a support vector machine for the maxi-mization phase. The di!erence between these two approachesis the choice of the hidden variables. In mi-SVM, the hid-

den variables are the labels of the individual feature vec-tors, whereas in MI-SVM, the hidden variable is a selectorvariable indicating which feature vector in the bag is posi-tive. There has also been research on building ensembles ofmultiple-instance learners and applying MIL to CBIR suchas the work of Zhou and Zhang [39] and Xu and Frank [35].

Rahmani et al. [26] introduced a new method to representthe image as a bag that is segmentation based but also intro-duces features to capture information about the relationshipof a segment to its four neighbors in the cardinal directions.As in the EM-DD algorithm [37], they use a variant of EM-DD which they call Ensemble EM-DD in which the averagelabel from all hypotheses returned from the multiple runs ofEM are used to rank the test images. Unlike other ensemble-based approaches, Ensemble EM-DD uses the hypothesis al-ready generated by a single execution of EM-DD as opposedto using bagging or boosting that would both require manyruns of EM-DD on di!erent training sets. One of the con-tributions of our work is our improvements to the EnsembleEM-DD algorithm.

Chen and Wang [3] consider the problem of image catego-rization. For this problem, one would typically have fairlylarge training sets and the image categories are generallyquite broad. They introduce DD-SVM that uses a one-against-the-rest strategy to handle the multi-class setting.To create the binary classifier for each category, DD-SVMfirst runs EM-DD from every feature vector of each positivebag to learn a set P of positive prototypes. Then it negatesall labels and runs EM-DD from every feature vector of eachpositive bag (the originally negative bags), to learn a set ofN negative prototypes. Then each bag in the training set isrepresented as a |P | + |N | dimensional feature vector wherethe value for dimension i is the weighted (using the learnedscale vector) Euclidean distance between the closest featurevector in the bag to the feature vector defining the ith di-mension. They then train a SVM to define a separator. Onedrawback of this approach is that there are several parame-ters that must be tuned for each data set including two thatare used to select the prototypes, and two used by the SVM.

Bi et al. [2] present 1-norm SVM which modifies DD-SVMin several key ways. First, EM-DD is not used, but rathereach feature vector in each positive bag is used as a posi-tive prototype and no negative prototypes are used. Second,the feature value in each new dimension uses an unweightedEuclidean metric. Finally, a 1-norm SVM is used since itreduces to solving a linear program which is more e"cient.The advantage of this approach is that when the trainingdata size is large (as common for image categorization), com-parable results can be obtained with significantly lower com-putation costs. However, this approach is not good whenthere are small data sets as one would expect for CBIR, andit does not perform any scaling and so all image featuresare treated equally. While this method removes the need totune the parameters used to select the instance prototypes,there are two parameters that must be tuned for the SVM.

Salient points based representation are also commonlyused in CBIR. In general, such methods rank images basedon the number of salient points that match between a sin-gle query image and the images in the repository. Althoughsalient points representations only depend on the portionsof the image around the salient points, when a single imagequery is used, such methods are global in that all salientpoints of the query image are used equally in the ranking.

Tian et al. [29] apply a wavelet-based salient points ex-traction method to CBIR. They define the color representa-tion of a salient point using the first moment (mean), second(variance) and third (skewness) central moment of the colorof the pixels in a 3!3 window centered at that salient point.They use the moments of the Gabor wavelet coe"cients ofthe pixels in a 9!9 window centered at that point as the thetexture representation of that salient point.

Scale Invariant Feature Transform (SIFT) [21] was intro-duced as a way to extract salient points that are invari-ant to many common image transforms. Mikolajczyk andSchmid [24] compared a variety of approaches of identify-ing salient points, and found SIFT to work best for imagematching. SIFT has also used in retrieval [33]. Variations ofSIFT, such as PCA-SIFT [15] and reduced SIFT [17] are alsoused in image retrieval. However, SIFT features and theirvariations are based on image gradients of the pixels in awindow around the salient point, so no color information isincluded in these features.

Wolf et al. [34] propose a salient point-based CBIR methodusing the response from the Gabor filter bank as the featurefor that point. Their representation captures only textureinformation for a salient point.

Some work consider combining the representation usingboth image segmentation and salient points. Lee and Ho [18]use a wavelet-based method to get the salient points, andalso segment the image to get the important regions assum-ing that main objects are located near the center of the ob-ject. They define texture features of an salient points as thewavelet coe"cients in a 3!3 window around that point, anddefine image feature for each region as the color histogramon that region. Although their retrieval is based on both thetexture features from the salient points and the color featurefrom segments, it uses the features from all salient points inthe image, so their technique is not a localized CBIR.

To address the limitations of current feature representa-tion methods and to improve retrieval performance, we pro-pose a localized CBIR system that uses multiple feature vec-tors combining region-based and salient point-based extrac-tion, and applys the Improved EM-DD as the local rankingmethod.

4. OUR NEW SALIENT POINTREPRESENTATION

A key contribution of our paper is the introduction of anew salient point representation for localized CBIR that isachieved using two orthogonal techniques. In the first, weuse an image segmentation to form a mask that limits thenumber of salient points in each segment while maintainingthe diversity of the salient points. In the second, we uselocal characteristics of a salient point to determine how tosplit the window into two sub-windows, and assign each sub-window features based on both it and its neighboring sub-window. We now describe these two methods in more depth.

4.1 SPARSE: Salient Point Reduction usingSegmentation

One of the di"culties of MIL is the ambiguity caused bynot knowing which feature vector(s) in a positive bag areresponsible for the label. Thus, the performance of MILalgorithms generally degrade as the size of the positive baggrows. An advantage of a region-based image representation

is that most images are segmented into a fairly small num-ber of objects (10-50) leading to relatively small bags. Manysalient points detection methods have been proposed [28, 25,11, 29, 21]. Most of these methods focus on finding the pointwhere there is significant change with respect to a chosenimage feature. Such representations provide a better viewof the local characteristics of an image than region-basedrepresentations. However, since salient points can gatherat locations of high texture and where the change of imagefeature is significant (such as edges), a very large number ofsalient points are generally needed to include salient pointsfor all objects that might be of interest. Even salient pointmethods that are deliberately designed to reduce the gather-ing, such as the wavelet-based method [29], still place manysalient points on some objects.

The SPARSE (Salient Points Auto-Reduction using SEg-mentation) image representation method that we proposelimits the number of salient points in each segment whilemaintaining the diversity needed for localized CBIR. SPARSEfirst applies a segmentation algorithm and a salient point de-tection algorithm to the image. The segmentation is thenapplied as a mask to reduce the number of salient points.Specifically, SPARSE keeps the k salient points in each seg-ment with the highest saliency value, and removes the oth-ers. If there are less than k salient points in a region, allare kept. In our implementation, k = 3. An important areaof future work is to perform experiments to determine thesensitivity to k, and develop methods to select the best k.

We use a wavelet-based salient-point detection methodlike that of Tian et al. [29]. More specifically, we use the Harrwavelet transform to get the image information at di!erentscales. Beginning at the coarsest level of wavelet coe"cients,we keep track of the salient points from level to level by find-ing the points with the highest coe"cients on the next finerlevel among those used to compute the wavelet coe"cientsat the current level. The saliency value of a salient point isthe sum of the wavelet coe"cients of its parent salient pointson all coarser scales. Although a wavelet-based method isused here, any salient point detection method can be appliedwith little modification.

The segmentation algorithm we use is a clustering-basedsegmentation method [8]. More specifically, each segmentis initially a single pixel, and then neighboring groups thatare most similar are merged into one group. This processis repeated until the desired number of segments is reached.The similarity measure we use is the Euclidean distance be-tween 6-dimensional feature vectors, with 3 color featuresand 3 texture features. For a region, its feature vector is theaverage features of all pixels in that region. Our methodscan be applied using any segmentation algorithm.

Throughout this paper, we represent a region (that couldbe a segment or the region around a salient point) using 3color features (Y, Cr, Cb) and 3 texture features (HL, LHand HH). Our methods could be applied using any other fea-tures such as the color moments and Gabor filter responses.

Since the segmentation is just used as a filter for reducingthe salient points, the performance of the CBIR system isnot as dependent on the exact boundaries of the segmen-tation since the representation for each salient point is justbased on the region around that salient point which is inde-pendent of the boundaries of the segment. In other words,the segmentation only a!ects the selection of the salientpoints but not their representation.

Examples of salient points detected using SPARSE areshown in Figure 2. For comparison, we also show the salientpoints detected by a classic wavelet-based salient points de-tection method (such as the Loupias method [20]), and theSIFT [21] method. We let both the SPARSE and wavelet-based methods detect 89 salient points from the “tea box”image, and 88 salient points from the “coke can” image.SIFT (which is run on an image 1/4 the size to reduce thenumber of salient points) has 126 points for the tea box, and118 salient points for the coke can. Results show that us-ing SPARSE the salient points gather at where the object iscomplex, but using wavelet-based method, the salient pointsgather at the edges. While the wavelet-based method doesreduce the number of salient points on the textured region(such as at the printed words on the calendar and tea box),SPARSE further reduces the number of salient points thatgather at textured regions.

4.2 VSWN: Our Salient Point RepresentationWe believe a better salient point representation can be de-

fined if the characteristics of the pixels around that salientpoint can be captured on a per-object basis, and if the spa-tial relationship of these pixels can be included. In thissection we describe the Variably-Split Window with Neigh-bor (VSWN) representation technique. We describe the twocomponents of VSWN individually.

4.2.1 Variably-Split WindowTraditional uses of salient points for CBIR assign features

to a salient point based on the image features of all pixelsin a window around that point. Since salient points areoften on the boundary of objects, the features assigned to asalient point often involve pixels from di!erent objects. Asillustrated in Figure 3, for an 8!8 window around a salientpoint detected on the edge of the tea box, there are pixelsfrom both tea box and calendar in the window. If we useall pixels in that window to compute features for the salientpoint, the features are for the combination of tea box andcalendar, which is inappropriate for localized CBIR becausethe object of interest might be just one of the object (e.g. thetea box). Furthermore, the background (e.g. the calendar)may not be relevant to the user. If we divide the window,we can better capture the color and texture of one of theindividual objects that borders the salient point.

For each salient point, VSWN dynamically determineshow to partition the pixels in the window around it into dif-ferent sub-windows. The local characteristics of the salientpoint are used to split each window either in the horizontal,vertical, or one of the two diagonal directions. Figure 4 il-lustrates these four possible splits. We use an 8!8 windowwith the salient point at its center, as shown in Figure 3.

We introduce the VSW (Variably Split Window) tech-nique to adaptively choose the best split. VSW applies awavelet transform on the pixels in the window, and measurethe average coe"cients in the HL (vertical), LH (horizontal),and HH (up-right diagonal). We also flip the window wherepixel p(i, j) have the value of the pixel p(i, windowWidth"j)to compute a flipped-HH coe"cient (up-left diagonal). Thesplit associated with each channel is shown in Figure 4. Weselect the split for the window as follows. If the LH and HLchannels have similar coe"cients, then we use the split as-sociated with the larger of the HH and flipped-HH channel.

Original image SPARSE Wavelet-based Method SIFT

Figure 2: Salient points detection with SPARSE, SIFT, and Wavelet-based Method.

Figure 3: An example of a split 8 ! 8 salient pointwindow (centered at the red pixel).

LH HL HH Flipped HH

Figure 4: Variably-Split Window.

Otherwise, we use the split based on the largest of the fourchannels since it is the dominant channel. While the bestsegmentation of the region is unlikely to be one of the foursplits considered, since it is such a small region, the selectedsplit serves as a su"ciently good approximation. If desired,we could further subdivide each sub-window.

4.2.2 Feature Representation with NeighborThe content desired by a user cannot always be defined by

a single object. For example, a flower is a set of petals ad-jacent to green leaves. There is a semantic gap between theinformation provided by a single segment and the semanticcontent the user perceives. Incorporating information aboutthe neighboring sub-window can reduce this gap.

Recall that we represent the two sub-windows for eachsalient point via 3 color features and 3 texture features.The feature vector for each sub-window is augmented withthe di!erence between the other sub-window’s value and itsvalue for each of the six features. We use the di!erence be-tween these feature values to allow for robustness againstglobal changes in the image, such as changes in brightnessand hues that result from di!erent lighting. We do not knowwhich sub-window might hold the object of interest, andwhich is the neighboring object. Thus we create two 12-dimensional feature vectors for each salient point: one foreach sub-window as the object of interest.

5. EM-DD AND ENSEMBLE EM-DDIn this section we describe the algorithms upon which Im-

proved EM-DD is based. We define a point-and-scaling hy-pothesis h = {t1, . . . , td, s1, . . . , sd} where tk is the featurevalue for dimension k and sk is a scale factor indicating theimportance of feature k.

The EM-DD algorithm [37] creates a point in scaling hy-pothesis. Each run of EM-DD starts with an initial guess ofa target point h and then repeatedly performs the followingtwo steps. In the first step (E-step), the current hypoth-esis h is used to pick one instance from each bag which ismost likely (given the generative model) to be the one re-sponsible for the label. In the second step (M -step), a two-step gradient search is used to find a new h that minimizesNLDD(h, D) (negative logarithm of the diverse density ofhypothesis h for data D which is a measure of likelihoodof h being the hypothesis given data D). From among themultiple-starts of EM, the point-and-scaling concept withthe minimum NLDD value is returned. Finally, a run ofEM is started from every positive point from five randomlyselected positive bags (or from all positive bags if there areless than five) with all scale factors set to 0.1. Assume thereare t runs of EM returning hypotheses h1, . . . , ht. EM-DDreturns argminihi.

There are several key changes the distinguish Ensemble-EMDD from EMDD [26]. First, along with starting at aninitial point from a randomly selected set of positive bags,di!erent initial scale factors for the weighting given to thesegment or its neighbors are used. Specifically the initialweights used for the segment itself is 100%, 80%, 60%, 20%(all 5 regions, equally weighted), and 0%. Also, instead ofreturning the single hypothesis that maximizes the diversedensity, Ensemble-EMDD ranks the images according to theaverage label produced by h1, . . . , hr. Finally, all initial scalefactors are adjusted based on the characteristics of the train-ing data and the floating point precision of the underlyingcomputer to reduce overflow and underflow problems in thegradient search.

6. OUR IMPROVED EM-DD ALGORITHMIn this section, we describe the most significant changes

introduced by our improved EM-DD algorithm. The mostsignificant change is the way in which the t hypothesis cre-ated by the t di!erent starts of EM are combined. ForCBIR applications, there are often several independent ways

to characterize which images the user desires. Several re-searchers have observed that for CBIR applications selectinga single hypothesis among h1, . . . , ht reduces the expressivepower of the resulting hypothesis [38, 3, 26]. In Section 6.1we discuss this aspect of Improved EM-DD.

We also make some other changes that we briefly discuss.Another important aspect of EM-DD that is the way inwhich it determines the importance of the features. Thefeature values from the feature vectors in positive bags pro-vides a good starting point for the hypothesis point in thegradient search. However, there is not a good way to de-termine the initial scale values and so equal values are usedfor each dimension. Still the range for features can varygreatly. For example color features typically range from 0to 256 whereas the values returned from the wavelet filterstypically used for texture are generally much smaller rang-ing from 0 to 5 (but can be quite large). Improved EM-DDovercomes this di"culty by normalizing all attributes basedon the training data, so that the range of values for all fea-tures on the training data is 0 to 1. The normalization factorfor each dimension is returned along with the hypotheses sothat the same normalization can be applied to the test data.While the test data can have data values in a di!erent rangeand thus their normalized values could be smaller than 0 orlarger than 1. But by performing this normalization, inpractice, all features begin with roughly equal importance.

6.1 Hypothesis SelectionWe describe several approaches to use the hypotheses

h1, . . . , ht returned from the t runs of EM used within Im-proved EM-DD. For ease of exposition, we use the followingnotation throughout this section. Without loss of gener-ality, we assume that the hypothesis are sorted based onthe NLDD in the order h1, . . . , ht. For hypothesis h, we letph(B) be the maximum label predicted for all feature vectorsin bag B. There are two options previously proposed.

minNLDD: This is the option used by the original DD andEM-DD algorithms. In this case each test bag B isranked according to ph1(B). That is, ph1 is computedfor all test bags and then they are ranked from thelargest to smallest value.

AvgAll: This is the option used by [38] and [26]. It rankseach test bag B according to

(ph1(B) + ph2(B) + · · · + pht(B))/t.

While in many cases combining all t hypotheses returnedby the gradient searches yields better performance than us-ing the top hypothesis, one would expect that there are a setof hypothesis that provide independent ways to characterizethe images of interest to the user, yet there are also somehypothesis that are the result of a bad starting point forEM that are only harming performance. Furthermore, onewould expect that the hypotheses with a low NLDD valueare the “good” ones and the hypotheses with high NLDDvalues should be excluded. This is the motivation behind thecombining method used by Improved EM-DD. We parame-terize Improved EM-DD by an integer ! where 1 # ! # t,and rank each test bag B according to

(ph1(B) + ph2(B) + · · · + ph! (B))/!.

This approach generalizes the prior work since when ! = 1it is equivalent to the minNLDD method, and when ! = t

it is equivalent to the AvgAll method. Currently, we arejust using a single value of ! . However, we believe thatadditional improvements can be obtained by using a tech-nique to adaptively select the best value of ! . For example,one could select all hypothesis that have an NLDD valuewith a multiplicative factor of h1. We are currently con-sidering methods to enable Improved EM-DD to select ! aspart of the learning process. Unlike other ensemble-basedapproaches, Improved EM-DD uses the hypothesis alreadygenerated by a single execution of EM-DD as opposed to us-ing bagging or boosting that would both require many runsof EM-DD on di!erent training sets.

7. EXPERIMENTAL RESULTSAs a measure of performance we have chosen to use the

area under the ROC curve [10]. The ROC (Receiver Oper-ating Characteristic) curve plots the true positive rate (i.e.the recall) as a function of the false positive rate. The areaunder the ROC curve (AUC) is equivalent to the probabil-ity that a randomly chosen positive image will be rankedhigher than a randomly chosen negative image. Unlike theprecision-recall curve, the ROC curve is insensitive to ra-tio of positive to negative examples in the image repository.Regardless of the fraction of the images that are positive,for a random permutation the AUC is 0.5.

We use the SIVAL data set that we obtained fromwww.cse.wustl.edu/$sg/accio. It includes 25 image cat-egories containing 1500 images. The categories consist ofcomplex objects photographed against 10 di!erent highlydiverse backgrounds. More specifically, there are six dif-ferent images taken for each object-background pair, and atotal of 1500 images in the database. (Figure 1 is an exampleimage of “JuliesPot”.) SIVAL emphasizes the task of Local-ized CBIR through nearly identical scenes which only varyby the localized target objects. It ensures there is one andonly target object in each image. For each object class thesame physical object was used in all scenes. However, thescenes are highly diverse and often complex. Furthermore,the objects may occur anywhere spatially in the image andalso may be photographed at a wide-angle or close up and atslightly di!erent orientations. In most image, the target ob-ject occupies less than 10%-15% of the image area but mayoccupy as much as 70%. Finally, the objects themselves arecomplex.

Unless otherwise mentioned the experimental results are30 fold with 95% confidence interval. The training data con-sists of 8 randomly selected positive images and 8 randomlyselected negtive images. We consider the following meth-ods to generate each feature vector in the MIL example:SPARSE, SIFT, Wavelet (the wavelet-based salient pointextraction method as used by the Loupias method), andSegmentation (segmented by IHS [40], used also as the fil-ter in SPARSE). When using either SPARSE or Wavelet,we also specify if VSWN is used by a notation followinga plus sign. For example, SPARSE+VSWN refers to ournew salient point extraction method that also uses VSWNto compute the feature vector for each salient points. Withthe exception of SIMPLIcity and Accio, unless otherwisespecified, the Improved EM-DD algorithm is used.

When converting the images into bags, we used the fol-lowing parameters. Segmentation breaks the image into 32segments, identical to Rahmani et al in Accio [26]. Waveletuses the top 100 salient points. SPARSE has up to 96 salient

Salient Point Extraction

0.4

0.5

0.6

0.7

0.8

0.9

1

GoldMedal

WD40Can

TranslucentBowl

SmileyFaceDoll

GreenTeaBox

CokeCan

SpriteCan

FabricSoftenerBox

CheckeredScarf

JuliesPot

BlueScrunge

AjaxOrange

CandleWithHolder

Banana

FeltFlowerRug

DirtyRunningShoe

DataMiningBook

GlazedWoodPot

CardboardBox

DirtyWorkGloves

StripedNotebook

RapBookApple

LargeSpoon

WoodRollingPin

AU

C

SPARSE

Wavelet

SIFT

Figure 5: Comparing Salient Point Methods.

points - up to 3 points for each of 32 segments. SIFT obtainsa variable number of salient points per image.

The average performance of these methods are shown inTable 1. Experimental results show that all three tech-niques we proposed improve retrieval performance. UsingSPARSE and VSWN together, we have a 11.0% increasein performance over Wavelet salient points. SPARSE alonecan improve the performance by 9.3% and VSWN alone canimprove the performance by 4.8%. Improved EMDD aloneperforms up to 10.4% better than Ensemble EMDD whenusing identical representations. More detailed comparisonsare given in the following sub-sections.

CBIR Method AUC

SPARSE+VSWN 81.6SPARSE 80.3VSWN 77.0Wavelet 73.5IHS Segmentation 81.8SIFT 77.9Accio! (IHS Segmentation) 74.6Accio! (SPARSE+VSWN) 73.8Accio! (Wavelet) 69.1SIMPLIcity 57.9

Table 1: Average AUC(%) over all SIVAL categoriesfor 30 random runs. Improved EMDD is used for allrepresentations except Accio! and SIMPLIcity.

7.1 Comparing Salient Point MethodsFigure 5 compares the three salient point extraction meth-

ods illustrated in Figure 2. Improved EMDD algorithm isused to generate these results.

In these experiments, both SPARSE and Wavelet use thesame salient point extraction and representation method (3color and 3 texture dimensions). The primary di!erence be-tween them is where the salient points are placed in the im-age. SIFT both uses a di!erent feature extraction method,placing the salient points di!erently, and uses a more com-plex feature representation.

SPARSE outperforms Wavelet in 23 of 25 categories, 16of which are statistically significant. So, by spreading outand reducing the number of salient points using a segmen-

New Salient Point Methods

0.4

0.5

0.6

0.7

0.8

0.9

1

GoldMedal

WD40Can

TranslucentBowl

SmileyFaceDoll

GreenTeaBox

CokeCan

SpriteCan

FabricSoftenerBox

CheckeredScarf

JuliesPot

BlueScrunge

AjaxOrange

CandleWithHolder

Banana

FeltFlowerRug

DirtyRunningShoe

DataMiningBook

GlazedWoodPot

CardboardBox

DirtyWorkGloves

StripedNotebook

RapBookApple

LargeSpoon

WoodRollingPin

AU

C

WaveletVSWNSPARSESPARSE VSWN

Figure 6: SPARSE and VSWN as components.

tation mask, SPARSE improves retrieval performance. Theuse of SPARSE can also improve runtime performance, byreducing the number of feature vectors per bag, and hencethe number of computations.

The SIFT feature vector has 128 dimensions which de-scribes the local gradient orientation histogram around asalient point. SIFT is a well established method for captur-ing and representing salient points. Results using SIFT weregenerated using 5 random selections of the training data (asopposed to 30) since the high dimensionality makes it verycomputationally intensive. SIFT performs 5.9% better thanWavelet over all categories. But SPARSE outperforms SIFTby 3.2% over all categories, despite its relatively simpler fea-ture representation of only 6 dimensions.

Figure 6 compares the e!ect of using SPARSE and VSWNto the standard wavelet-based salient points on the SIVALdatabase. Adding VSWN to Wavelet leads to an 4.8% im-provement when averaged over all categories. These im-provements are statistically significant in 9 categories. AddingSPARSE to the Wavelet leads to an 9.3% improvement whenaveraged over all categories. When both SPARSE and VSWNare added to the Wavelet, on average there is a 11.0% in-crease in performance.

The results show that both SPARSE and VSWN, as indi-vidual techniques to improve image representation, can helpimprove the retrieval performance. When used together,they can improve the performance even further.

7.2 Comparing Learning AlgorithmsFigure 7 compares the e!ect of using Improved EMDD

versus the Ensemble EMDD algorithm used in Accio. Im-proved EMDD performs better on both the Wavelet salientpoints as well as the SPARSE+VSWN representation. Com-paring Improved EMDD to Ensemble EMDD, performanceimproves 6.4% on the Wavelet representation and 10.4% forSPARSE+VSWN representation.

SPARSE+VSWN gains more from Improved EMDD thanWavelets do. We attribute this to the use of only the best hy-potheses instead of all hypotheses, which is more importantwhen multiple initial weights are used due to the presenceof neighbors, as we do in SPARSE+VSWN. Multiple initialweights multiplicatively increase the number of hypothesespresent. Ensemble EMDD cannot handle this increase inhypotheses as well as Improved EMDD can.

Algorithm Comparison

0.4

0.5

0.6

0.7

0.8

0.9

1

GreenTeaBox

GoldMedal

WD40Can

SmileyFaceDoll

TranslucentBowl

CokeCan

JuliesPot

CheckeredScarf

FeltFlowerRug

BlueScrunge

SpriteCan

AjaxOrange

FabricSoftenerBox

Banana

GlazedWoodPot

DataMiningBookApple

DirtyRunningShoe

CandleWithHolder

DirtyWorkGloves

CardboardBox

StripedNotebook

WoodRollingPin

RapBook

LargeSpoon

AU

C

SPARSE VSWN, Improved EMDDSPARSE VSWN, Accio! AlgWavelet, Improved EMDDWavelet, Accio! Alg

Figure 7: Learning Algorithm Comparison.

7.3 Comparisons Between CBIR SystemsWe compare 3 systems: SPARSE+VSWN using the Im-

proved EMDD algorithm, Accio, and SIMPLIcity, as shownin Figure 8. The comparison with Segmentation using im-proved EMDD will be detailed in Section 7.4.

The SIMPLIcity and Accio systems perform significantlyworse than the SPARSE+VSWN using the Improved EMDDalgorithm. The SIMPLIcity system is shown here as a ref-erence point. For a full comparison of SIMPLIcity to Accio,please see [26]. Compared to Accio, SPARSE+VSWN withImproved EM-DD performs 9.4% better over all categories.It performs better in 21 of 25 categories, and statisticallybetter in 14 of 25 categories. Furthermore, it has signif-icantly faster runtime than Accio, which on average tookover 25 times longer to complete. The improvement in runtime is due to the reduction in dimensionality and the re-duction of final hypotheses. Segmentation with Neighborsuses 30 dimensions per point, where as SPARSE+VSWNuses only 12. The number of final hypotheses that Acciouses to evaluate each image in the database is over 20,000in most categories, whereas SPARSE+VSWN uses only 75.

In other words, when used together as a system, our tech-niques can achieve better performance than the current CBIRsystems shown here.

7.4 Comparison with Segmentation withNeighbors

Figure 8 also compares the SPARSE+VSWN and the Seg-mentation with Neighbors representation of Accio both onthe Improved EMDD algorithm. We see similar performancein 17 of 25 categories. In 5 categories, Segmentation is stat-ically better; and in 3 categories, SPARSE+VSWN is sta-tistically better. Overall, their results are comparable.

SPARSE+VSWN has several advantages over Segmenta-tion with Neighbors though. It significantly reduces runningtime by reducing the dimensionality from 30 to 12. Moreimportantly, VSWN uses only one neighbor as opposed tothe four neighbors (in the four cardinal directions) of thesegmentation-based representation. Since the four neigh-bors are oriented to be represent north, east, south, andwest, this restricts matching of a desired object to only otherobjects that are similarly oriented. In contrast, the variably-splitting technique of VSWN creates only one neighbor which

CBIR Systems

0.4

0.5

0.6

0.7

0.8

0.9

1

GreenTeaBox

GoldMedal

WD40Can

SmileyFaceDoll

TranslucentBowl

CokeCan

JuliesPot

CheckeredScarf

FeltFlowerRug

BlueScrunge

SpriteCan

AjaxOrange

FabricSoftenerBox

Banana

GlazedWoodPot

DataMiningBookApple

DirtyRunningShoe

CandleWithHolder

DirtyWorkGloves

CardboardBox

StripedNotebook

WoodRollingPin

RapBook

LargeSpoon

AU

C

Segmentation, Improved EMDD

SPARSE VSWN, Improved EMDD Accio!SIMPLIcity

Figure 8: Comparisons Between CBIR Systems.

is both mirror invariant and rotation invariant. The SIVALdata set, is a good data set for using the 4 cardinally-orientedneighbors being used by the segmentation-based approach.Although there are often changes in the perspective of theobject, there is seldom rotation in the plane of the image norare target objects ever mirrored. The three categories thatexperience significant rotation in the ranges of 90! and 180!

are WD40Can and DirtyWorkGloves, and WoodRollingPin.Both the WD40Can and DirtyWorkGloves, benefit greatlyfrom the use of salient points, with an average percentageimprovement of 9.3%. The WoodRollingPin performs iden-tically poorly for both SPARSE+VSWN and Segmentationwith Neighbors achieving an AUC of 0.65, but performsmuch better for SIFT achieving an AUC of 0.72. The poten-tial of using additional features to improve SPARSE+VSWNis shown by the strong improvement in SIFT. Although thecurrent salient points encode only the same information asthe segmentation-based method, salient points can be en-coded with a multitude of additional features not easily de-rived from segmentation methods, such as the orientationhistogram used by SIFT.

Additionally, salient points by their nature can capturemuch finer detail in the image than segmentation. The onlyway for a segmentation method to capture the same detailwould be to highly over-segment the image which we believewould hurt recognition.

8. CONCLUSIONSIn this paper we introduce the SPARSE technique, to se-

lect a small number of salient points by using a segmentationof the original image as a filter to reduce the total number ofsalient points while still maintaining diversity. We also intro-duce the VSWN salient point representation which generatesa feature vector to represent each salient point by applyinga window that can be dynamically split into a central targetand its neighbor. Results demonstrate that both techniquesgreatly improve retrieval accuracies. Finally, we introducedan improved EM-DD algorithm to learn the hypothesis usedto rank the images. This algorithm improved both the re-trieval performance on both the segmentation-based repre-sentation and the salient-points based representation.

Given additional image set that include strong rotationsor occlusions of the objects in the image, we believe salientpoints may be able to work much better than the segmenta-

tion based representation because of the rotation invariantand mirror invariant properties of salient points based rep-resentation.

Future work includes applying SPARSE and VSWN tosalient points with more descriptive features such as those inSIFT. Additionally, we intend to develop a methodology forautomatically selecting the number of salient points per seg-ment in SPARSE, according to local characteristics. Finally,we plan on improving the algorithm by developing bettermethods of hypothesis selection and implementing prepro-cessing techniques that will reduce the number of startingpoint and hence reduce the runtime. We also plan to per-form sensitivity analysis for the number of salient pointskept in each segment, and the parameter used within thevariable split portion of VSWN.

9. ACKNOWLEDGMENTSWe would like to thank John Smith and the anonymous

reviewers for their valuable feedback. This material is basedupon work supported by the National Science Foundationunder Grant No. 0329241.

10. REFERENCES[1] S. Andrews, T. Hofmann, and I. Tsochantaridis. Multiple

instance learning with generalized support vector machines.Artificial Intelligence, pages 943–944, 2002.

[2] J. Bi, Y. Chen, and J. Wang. A sparse support vector machineapproach to region-based image categorization. IEEE Conf. onComputer Vision and Pattern Recognition, pages 1121–1128,2005.

[3] Y. Chen and J. Wang. Image categorization by learning andreasoning with regions. Journal of Machine LearningResearch, pages 913–939, 2004.

[4] I. Cox, M. Miller, S. Omohundro, and P. Yianilos. PicHunter:Bayesian relevance feedback. Int. Conf. on PatternRecognition, pages 361–369, 1996.

[5] A. Dempster, N. Laird, and D. Rubin. Maximum likelihoodfrom incomplete data via the em algorithm. Journal of theRoyal Statistics Society, pages 1–38, 1977.

[6] T. Dietterich, R. Lathrop, and T. Lozano-Perez. Solving themultiple-instance problem with axis-parallel rectangles.Artificial Intelligence, pages 31–37, 1997.

[7] M. Fischler and R. Bolles. Random sample consensus: aparadigm for model fitting with applications to image analysisand automated cartograpy. Comm. ofthe ACM, 24:381–395,1981.

[8] D. Forsyth and J. Ponce. Computer Vision: A ModernApproach. Prentice Hall, 2003.

[9] T. Gevers and A. Smeulders. Image search engines: Anoverview. In G. Medioni and S. B. K. (Eds.), editors,Emerging Topics in Computer Vision. Prentice Hall, 2004.

[10] J. Hanley and B. McNeil. The meaning and use of the areaunder a receiver operating characteristic (ROC) curve.Radiology, 43:29–36, 1982. 1982.

[11] C. Harris and M. Stephens. A combined corner and edgedetector. Alvey Vision Conference, 1988.

[12] X. He, W.-Y. Ma, O. King, M. Li, and H. Zhang. Learning andinferring a semantic space from user’s relevance feedback forimage retrieval. ACM Multimedia, 2002.

[13] X. Huang, S.-C. Chen, M.-L. Shyu, and C. Zhang. Userconcept pattern discovery using relevance feedback andmultiple instance learning for content-based image retrieval.8th Int. Conf. on Knowledge Discovery and Data Mining,pages 100–108, 2002.

[14] J. W. J. Li and G. Wiederhold. IRM: Integrated regionmatching for image retrieval. 8th ACM Int. Conf. onMultimedia, pages 147–156, 2000.

[15] Y. Ke and R. Sukthankar. PCA-SIFT: A more distinctiverepresentation for local image descriptors. IEEE Conf. onComputer Vision and Pattern Recognition, 2004.

[16] S. Kim, S. Park, and M. Kim. Central object extraction forobject-based image retrieval. Int. Conf. on Image and VideoRetrieval, pages 39–49, 2003.

[17] L. Ledwich and S. Williams. Reduced sift features for imageretrieval and indoor localisation. Australian Conf. on Roboticsand Automation, 2004.

[18] H.-K. Lee and Y.-S. Ho. A region-based image retrieval systemusing salient point extraction and image segmentation. LectureNotes in Computer Science: Advances in MultimediaInformation Processing - PCM 2002: Third IEEE PacificRim Conference on Multimedia, 2002.

[19] F. Long, H. Zhang, and D. Feng. Fundamentals ofcontent-based image retrieval. In D. Feng, W. Siu, and H. Z.(Eds.), editors, Multimedia Information Retrieval andManagement- Technological Fundamentals and Applications.Springer, 2003.

[20] E. Loupias. Salient points detection using wavelet transform,http://telesun.insa-lyon.fr/ loupias/points/demo.html.

[21] D. G. Lowe. Distinctive image features from scale-invariantkeypoints. Int. Journal of Computer Vision, 2004.

[22] O. Maron and T. Lozano-Perez. A framework formultiple-instance learning. Neural Information ProcessingSystems, 1998.

[23] O. Maron and A. Ratan. Multiple-instance learning for naturalscene classification. 15th Int. Conf. on Machine Learning,pages 341–349, 1998.

[24] K. Mikolajczyk and C. Schmid. A performance evaluation oflocal descriptors. Computer Vision and Pattern Recognition,2003.

[25] H. P. Moravec. Towards automatic visual obstacle avoidance.5th Int. Joint Conf. on Artifical Intelligence, 1977.

[26] R. Rahmani, S. Goldman, H. Zhang, J. Krettek, and J. Fritts.Localized content-based image retrieval. ACM Workshop onMultimedia Image Retrieval, pages 227–236, 2005.

[27] Y. Rui, T. Huang, M. Ortega, and S. Mehrotra. Relevancefeedback: A power tool in interactive content-based imageretrieval. IEEE Transactions Circ. Sys. Video Technology,pages 644–655, 1998.

[28] C. Schmid, R. Mohr, and C. Bauckhage. Evaluation of interestpoint detectors. Int. Journal of Computer Vision,37(2):151–172, 2000.

[29] Q. Tian, N. Sebe, M. S. Lew, E. Loupias, and T. S. Huang.Image retrieval using wavelet-based salient points. Journal ofElectronic Imaging, Special Issue on Storage and Retrieval ofDigital Media, 2001.

[30] Q. Tian, Y. Wu, and T. Huang. Combine user definedregion-of-interest and spatial layout for image retrieval. ICIP,2000.

[31] P. Torr and A. Zisserman. MLESAC: a new robust estimatorwith application to estimating image geometry. Comput.Vision Image Understanding, 78:138–156, 2000.

[32] J. Wang, J. Li, and G. Wiederhold. SIMPLIcity:semantics-sensitive integrated matching for picture libraries.IEEE Trans. on Pattern Analysis and Machine Intelligence,pages 947–963, 2001.

[33] J. Wang, H. Zha, and R. Cipolla. Combining interest pointsand edges for content-based image retrieval. IEEE Int. Conf.on Image Processing, 2005.

[34] C. Wolf, J.-M. Jolion, W. Kropatsch, and H. Bischof. Contentbased image retrieval using interest points and texturefeatures. IEEE Int. Conf. on Pattern Recognition, 2000.

[35] X. Xu and E. Frank. Logistic regression and boosting forlabeled bags of instances. In H. Dai, R. Srikant, and C. Zhang,editors, Proc 8th Pacific-Asia Conference on KnowledgeDiscovery and Data Mining, volume 3056 of LNAI, pages272–281, Sydney, Australia, 2004. Springer.

[36] C. Yang and T. Lozano-Perez. Image database retrieval withmultiple instance techniques. 16th Int. Conf. on DataEngineering, pages 233–243, 2000.

[37] Q. Zhang and S. Goldman. EM-DD: an improvedmultiple-instance learning technique. Neural InformationProcessing Systems, 2001.

[38] Q. Zhang, S. Goldman, W. Yu, and J. Fritts. Content-basedimage retrieval using multiple instance learning. 19th Int.Conf. on Machine Learning, pages 682–689, 2002.

[39] Z. H. Zhou and M. L. Zhang. Ensembles of multi-instancelearners. 14th European Conf. on Machine Learning, 2003.

[40] H. Zhang, J. Fritts and S. Goldman, An Improved Fine-grainHierarchical Method of Image Segmentation. WashingtonUniversity CSE Technical Report, 2005.