Image segmentation based on merging of sub-optimal segmentations

www.elsevier.com/locate/patrec

Pattern Recognition Letters 27 (2006) 1105–1116

Image segmentation based on merging of sub-optimal segmentations

Juan C. Pichel *, David E. Singh, Francisco F. Rivera

Dept. Electronica e Computacion, Universidade de Santiago de Compostela, Galicia, Spain

Received 29 April 2004; received in revised form 2 November 2005Available online 8 February 2006

Communicated by M.A.T. Figueiredo

Abstract

In this paper a heuristic segmentation algorithm is presented based on the oversegmentation of an image. The method uses a set ofdifferent segmentations of the image produced previously by standard techniques. These segmentations are combined to create the over-segmented image. They can be performed using different techniques or even the same technique with different initial conditions. Based onthis oversegmentation a new method of region merging is proposed. The merging process is guided using only information about thebehavior of each pixel in the input segmentations. Therefore, the results are an adequate combination of the features of these segmen-tations, allowing to mask the negative particularities of individual segmentations. In this work, the quality of the proposal is analyzedwith both artificial and real images using a evaluation function as case of study. The results show that our algorithm produces high qual-ity global segmentations from a set of low quality segmentations with reduced execution times.� 2006 Elsevier B.V. All rights reserved.

Keywords: Segmentation; Region-merging heuristic; Segmentation evaluation

1. Introduction

The problem of the segmentation of images generallyimplies the partitioning of an image into a number ofhomogeneous regions, so that joining two of these neigh-boring regions gives rise to another heterogeneous region.Many ways of defining the homogeneity of a region exist,depending on the context of the application. Moreover, awide variety of methods and algorithms are available todeal with the problem of the segmentation of images (Fuand Mui, 1981; Haralick and Shapiro, 1985; Pal and Pal,1993). These methods can be broadly classified in fourcategories (Zhu and Yuille, 1996):

• Edge-based techniques.• Region-based techniques.

0167-8655/$ - see front matter � 2006 Elsevier B.V. All rights reserved.

doi:10.1016/j.patrec.2005.12.012

* Corresponding author. Tel.: +34 981563100x13564; fax: +34 981528012.E-mail addresses: [email protected] (J.C. Pichel), [email protected]

(D.E. Singh), [email protected] (F.F. Rivera).

• Deformable models.• Global optimization approaches.

The edge-based techniques are based on informationabout the boundaries of the image. Therefore, they try tolocate the points in which abrupt changes occur in thelevels of some property of the image, typically brightness(Canny, 1986; Rosenfeld and Kak, 1982). On the otherhand, those methods that use spatial information of theimage (e.g. color or texture) to produce the segmentedimage fit into the region-based techniques (Chen et al.,1992; Sahoo et al., 1988). These methods depend on theconsistency of some relevant property in the differentregions of the image. The deformable models are basedon curves or surfaces defined within an image that movesdue to the influence of certain forces. They can be classifiedin various groups, principally snakes, deformable templatesand active contours (Blake and Isard, 1998; Kass et al.,1988). All of these techniques avoid the use of a global cri-terion when segmenting the image, which is contrary to the

mailto:[email protected]



Image

Segmentation Segmentation Segmentation

1 2 N

Partial Segmentations

Combining

Oversegmented Image

Merging

Segmented Image

Info

rmat

ion

Fig. 1. Functional scheme of the proposed algorithm.

1106 J.C. Pichel et al. / Pattern Recognition Letters 27 (2006) 1105–1116

global optimization approaches (Geman and Geman, 1984;Kanungo et al., 1994).

As a consequence, several solutions can be obtainedwhen applying different segmentation techniques to thesame image due to the wide variety of segmentationalgorithms available for any context or application. Theresults of applying different techniques are usually dis-similar and normally the aforementioned solutions donot indicate evidently which of these methods performsa better segmentation of the image, even when the appli-cation context is well defined. Moreover, in some circum-stances, it could be useful to combine algorithms withdifferent goals if they are complementary. First, if theobjective of the segmentation is not defined in a preciseway. And second, when the objective can be expressedas a combination of several simple goals. For example,in order to detect blue and red objects it is useful to con-sider some algorithms to detect red objects and others todetect blue ones. So the information extracted from seg-mentation algorithms with different goals should not bealways considered as contradictory.

Based on this, we propose the idea of using various seg-mentations of the same image to create a final segmentedimage. This will adequately combine the characteristics ofthese segmentations and the negative particularities of eachof the segmentations can be masked by the others, so thatthe result is not dependent on the selection of a particularalgorithm or on some initial conditions. In this way, theresults obtained will be unified in one unique solutionand this will have an effect on the quality of the final seg-mentation. Each of these segmentations, which from nowon will be called partial segmentations, can be performedusing different techniques or even, the same technique witha different parametrization or with different initialconditions.

In Fig. 1 the functional scheme of the proposed algo-rithm is shown. To obtain a final solution, an overseg-mented image is created from partial segmentations.Afterwards, a region-merging algorithm is applied in sucha way that those regions that are similar, according to cer-tain criterion, are merged (Zhu and Yuille, 1996; Beaulieuand Goldberg, 1989; Shafarenko and Petrou, 1997). Theproposed criterion used by our algorithm is based exclu-sively on the information collected from the partial seg-mentations in the oversegmented image creation process.In addition, our approach is based on the introduction ofa repulsing force between neighboring regions that mea-sures the tendency of these to remain separated. The struc-ture of this work is as follows. In Section 2 the concept ofshadowed zones is introduced and how we can obtain theoversegmented image from the partial segmentations isexplained. In Section 3 the concept of the force of repulsionbetween regions that indicates the tendency of two regionsto merge or remain separated in the final segmentation isintroduced. Section 4 describes the merging process, whileSection 5 shows the results using, as case of study, a seg-mentation evaluation function as stopping criterion of

the merging process. Finally, Section 6 presents the mainconclusions of this work.

2. Generation of the oversegmented image

The segmentation of an image can be considered as alabeling problem, in the sense that those pixels that belongto the same region have been assigned the same label,which can be specified in terms of a set of states or a setof labels. Let H be a discrete set with m states, H ={1,2, . . . ,m} and let L be the set of labels. In our case, astate represents a region of the Euclidian space of the image(set of pixels). Labeling is not more than a function thatassigns one of the labels of L to each of the states of H.

Generally, the objective of the segmentation algorithmsbased on regions is the labeling of all the pixels of theimage. In many cases, in the final stages of the process, thissituation causes pixels to be included in regions from whichthey have very low similarity levels, thereby creatingregions with low homogeneity. To avoid this effect we pro-pose the inclusion of a specific threshold as a possible solu-tion. This threshold would be such that those pixels with alow degree of similarity with respect to the target regionare not included in it, remaining unlabeled. The set of un-labeled pixels will be called shadowed zones. Therefore, apixel, after the partial segmentation, can be labeled andthus belongs to a region, or it can be included in a shad-owed zone.

J.C. Pichel et al. / Pattern Recognition Letters 27 (2006) 1105–1116 1107

As an example, we checked the effect of using shadowedzones in a test image, performing the partial segmentationsusing the seeded region growing algorithm (SRG) intro-duced by Mehnert and Jackway (1997). The process beginswith a set of pixels, called seeds, which will be the basis forthe determination of the regions. It continues including,incrementally, the pixels on their boundaries according tosome given criterion to measure the similitude. Mehnertand Jackway use as criterion the difference between thegrey intensity of the pixel under consideration and the aver-age of the intensities of the pixels that already form part ofthe region. Fig. 2(a) shows the boundaries of three differentpartial segmentations of the image Test3 by using 20 seedsplaced randomly. In this type of algorithm, the initial num-ber of seeds determines the number of regions of the seg-mented image. Moreover, the quality of the result of thealgorithm depends in a great extend on the position ofthe seeds. As we have mentioned, the algorithm is forcedto label all the pixels of the image. This behavior becomesmanifest in these images, in which some regions includepixels with very low similitude. The quality of these seg-mentations is poor. By introducing a threshold in the levelof similitude, shadowed zones are created in the image.Three partial segmentations, using the same seeds as

Fig. 2. Examples of partial segmentations of image Test3 applying the SRGThe oversegmented image produced by the combination of the partial segmen

Fig. 2(a), with shadowed zones represented in grey, areshown in Fig. 2(b). Note that, with the use of shadowedzones, the number of correct regions detected by thesegmentation increases significantly with respect to the seg-mentation using the original algorithm. Despite of it, noneof these segmentations achieve the correct result.

Our proposal generates an oversegmented image as acombination of various partial segmentations in whichshadowed areas can exist. There are other segmentationtechniques that create an oversegmented image such aswatershed algorithms (Haris et al., 1998). The oversegmen-tations obtained from these algorithms can be used as inputof our algorithm, but not as unique oversegmented image.That is because, as we detail later, we need to collect, fromdifferent partial segmentations, the information that willguide the merging process. In our algorithm an operationfor intersecting all the partial segmentations is performedin such a way that those pixels that belong to the sameregion in all the partial segmentations remain united inone of the regions of the oversegmented image. Fig. 3shows a simple example of the creation of an over-segmented image formed, in this case, from three partialsegmentations. The first partial segmentation presents ashadowed zone and two regions, whilst in each of the other

algorithm (a) without shadowed zones and (b) using shadowed zones.tations of (b) is shown in (c).

Region B

Region D

Region E

Region C

Region A

0

1

2

3

4

0

1 2 3

4

Shadowed Zone

Fig. 3. Example of the generation of an oversegmented image from three partial segmentations of 5 · 5 pixels with three regions each one. Theoversegmented image presents five regions.


two segmentations there are three regions and no shad-owed zones. In this example the oversegmented imageconsists of five regions.

If shadowed zones exist in some of the partial segmenta-tions, they are considered as any other region in the gener-ation of the oversegmented image, although later, in themerging algorithm, they will be treated differently. Anotherexample is shown in Fig. 2(c). It shows the oversegmentedimage created from the partial segmentations of Fig. 2(b).

3. Force of repulsion between regions

One important contribution of our proposal is that allthe relevant information to obtain the final segmentedimage is obtained exclusively from the different partial seg-mentations, both for creating the oversegmented image andfor applying the subsequent region-merging algorithm. Itshould be noted that this strategy does not take intoaccount local characteristics such as size, shade of averagegrey intensities, etc. Based on the results obtained for allthe pixels of the image in each of the partial segmentations,global conclusions are obtained with respect to them.Without losing generality, in our algorithm we have chosena second-order neighborhood scheme, so that up to eightneighbors are defined for each pixel. In each partial seg-mentation we can find any pair of neighboring pixels inone of the following four different states:

1. Both belong to same labeled region.2. Both belong to different labeled regions.3. Both belong to shadowed zones.4. One belongs to a region and the other to a shadowed

zone.

We define a function to efficiently collect this informa-tion that adds up each one of these possible states. This

function summarizes the state of any pair of neighboringpixels of the image. Thus, over N partial segmentations,we can define the following counters for any pair of neigh-boring pixels i and j:

• n1(i, j)! Number of partial segmentations in which theyhave been in the same labeled region.

• n2(i, j)! Number of partial segmentations in whichboth have been in a shaded region.

• n3(i, j)! Number of partial segmentations in which theyhave been in different labeled regions.

• n4(i, j)! Number of partial segmentations in which onlyone of them has been in a shaded region.

It can be easily proven that the following properties arefulfilled:

nkði; jÞ ¼ nkðj; iÞ; being k ¼ 1; 2; 3 or 4 ð1Þn1ði; jÞ þ n2ði; jÞ þ n3ði; jÞ þ n4ði; jÞ ¼ N ð2Þ

In this way, those pixels i and j that belong to the sameregion in the oversegmented image should verify:

n3ði; jÞ ¼ n4ði; jÞ ¼ 0 ð3Þ

Based on these counters, we define a force of repulsionbetween two neighboring pixels i and j that measures thetendency of these pixels to be or not in the same region.We propose for this force a linear combination of theabove counters, given through the following equation:

fij ¼ �C1n1ði; jÞ � C2n2ði; jÞ þ C3n3ði; jÞ þ C4n4ði; jÞ ð4Þ

where Cn are parameters to weight each of the situations inwhich two neighboring pixels could be found. Therefore,two different components can be identified in Eq. (4): onone hand an attractive component given by term�C1n1(i, j) � C2n2(i, j) that measures the tendency of these

C

E

A

B D -1-1

3 1

-1

-1

0

3

3

Fig. 4. Region adjacency graph of the oversegmented image of Fig. 3.Each region is represented by a node, and between two nodes there is anedge if the regions are neighbors. The function used to assign weights tothe edges is the force of repulsion FRS with C1 = C2 = C3 = C4 = 1.


pixels to belong to the same region, and on the other, arepulsive component given by term C3n3(i, j) + C4n4(i, j)that measures the opposite. According with this equationwe can see that the lesser the force fij, the greater is thetendency for these pixels to belong to the same region.Additionally, we have that the pixels that belong to thesame region always verify that fij < 0.

For instance, let us consider the force of repulsion fij

among some pixels of the oversegmented image in Fig. 3.To simplify this example, the weight parameters of Eq.(4) are C1 = C2 = C3 = C4 = 1. The counters for the neigh-boring pixels (0,1) and (0,2), are n1 = 3, n2 = n3 = n4 = 0,and therefore, we obtain a force of repulsion fij = �3.For pixels (2,1) and (2, 2), we have n1 = n2 = 0, n3 = 2,n4 = 1, that causes an attraction force of fij = 3. Finally,for pixels (2, 3) and (2, 4) have n1 = 2, n3 = 1, n2 = n4 = 0,and fij = �1.

Next we show an analysis to determine the way theseparameters are related. C1 weights the tendency of a pairof pixels to be in the same region. Initially, we considerthe relationship between C1 and the parameter that weightsthe situation in which two pixels belong to different regions(C3). At first sight it seems adequate that both parametersmust have the same value because the situations that theyrepresent seem symmetrically opposite. Nevertheless, aswe have mentioned, one of the principal drawbacks ofmany segmentation algorithms, specially the region basedalgorithms such as SRG, is that two seeds placed withinthe same object in the image produce two different regionsin the segmented image although it is only one real region.For the merging algorithm to deal with this problem wemust reward, in some way, that two pixels belong to thesame region in the oversegmented image. So we proposethat C1 P C3.

Respect to the parameter that weights the situationwhen two pixels belong to a shadowed zone (C2), these pix-els are not treated by the partial segmentation algorithm.Note that using shadowed zones the dependence of theregion based algorithm with the placing of the seedsdecrease meaningfully. So, the fact that two pixels belongto a shadowed zone in the partial segmentation, does notbring enough information to decide if these pixels mustbelong to the same or to different regions, i.e., C2 mustbe small compared to the other parameters.

Finally, the parameter that weights the situation inwhich one pixel belongs to a shadowed zone and the otherto a region (C4), provides important information becausethe partial segmentation decided not to include it in alabeled region. Therefore, these pixels must belong to dif-ferent regions to avoid the creation of low homogeneityregions. So C4 P C1.

According to all these topics, we summarize the relation-ships between the parameters as

C4 P C1 P C3 P C2 ð5Þ

Now, we will introduce several definitions. Let R be a re-gion in the oversegmented image, and let S be a neighbor-

ing or adjacent region to it. We define the set of pixels of R

neighbors of S as

V R;S ¼ fi 2 Rj9j 2 S; i and j are neighborsg ð6ÞWe would say that S and R are neighbors if VR,S 5 ;.

For each i2VR,S, we define the associated set of itsneighbors belonging to S, as

UiR;S ¼ fj 2 Sji and j are neighbors, and i 2 V R;Sg ð7Þ

We define the force of repulsion between neighboring regions

as the average of the forces of repulsion of all the neighbor-ing pixels belonging to each one of those regions. If we de-note k Æk the cardinality operator, we can formalize theconcept by the following equation:

F RS ¼ F SR ¼

Xi2V R;S

Xj2Ui

R;S

fij

� �X

i2V R;SkU i

R;Skð8Þ

4. Region-merging algorithm

The structure of the data that we have used for represent-ing the partitions of the oversegmented image is a regionadjacency graph (RAG) (Haris et al., 1998). The RAG ofa segmentation of K regions is defined as a weighted undi-rected graph, G = (V,E), where V = {1, 2, . . . ,K} is the setof nodes and E � V · V is the set of edges. Each region isrepresented by a node, and between two nodes R,S 2 V

there is an edge (R,S) if the regions are neighbors. Fig. 4shows the RAG of the oversegmented image of Fig. 3. Aweight is assigned to each edge of the RAG, so that thosenodes joined by the lesser (or greater) weighted edge,depending on its definition, will be the regions that are can-didates for merging. In our case, the function used to assignweights to the edges is the force of repulsion FRS given byEq. (8), and therefore, the regions that are candidates formerging are those joined by the edge with least weight. Inthe example of Fig. 4, C1 = C2 = C3 = C4 = 1.

Using the RAG as input, we propose an iterative heuris-tic to deal with the problem of merging, so that one merg-ing is performed in each of the iterations, based on the

Fig. 5. Pseudo-code of the merging algorithm.


weight of the edges. A data structure adequate for storingthe weights is a queue, which can be implemented using aheap (Knuth, 1973). All the edges of the RAG are storedin the heap according to the weights, so that the first edgealways has the smallest weight. Evaluating the heaprequires O(kEk). In each iteration of the merging algo-rithm, the pair of regions that have the smallest weightare merged. One of the most useful properties of this iter-ative algorithm, as we show later, is that the weight ofthe edge that occupies the first position of the heap alwaysgrows.

Given the RAG of an initial partition of K regionsdenoted as (K � RAG), and a heap of its edges, theRAG of the partition K � n is obtained using the mergingalgorithm described in the pseudo-code of Fig. 5. TheK � RAG corresponds to the initial partition of the image,which in our case, is the oversegmented image. Subse-quently, an iterative process is applied, in which, in theith iteration, the two regions with the smallest weight aremerged. Once they have been merged, the list of edges isupdated, and the (K � i � 1) � RAG is obtained.

From the algorithm, it is inferred that n iterations areneeded to obtain the (K � n) � RAG. Therefore, one ofthe problems posed by this strategy is to establish the bestvalue of n, i.e., the best number of regions of the final seg-mented image. Different alternatives can be used. Forexample, using the property of the growing value of thefirst term of the heap, a certain threshold can be selectedto stop the iterations when the value of the edge at thetop of the heap exceeds it. The main drawback of thisapproach is that it is not evident to determine a goodthreshold a priori. Another approach is the use of a methodthat allows a numerical evaluation of the segmentation,and chooses the best threshold according to some selectedvalidation criterion.

5. A case of study: evaluation function Q

Most of the results obtained by segmenting an image areevaluated visually and qualitatively. The principal problemis searching for an uniform criterion to evaluate the results.Different methods for evaluating the segmentation ofimages have been published (Levine and Nazif, 1985;Zhang, 1996). For instance, the evaluation function Q pro-posed by Borsotti et al. (1998) for color images. This func-tion is a variant of that proposed by Liu and Yang (1994).

One of its main advantages is that do not require any exter-nal parameter. Specifically, function Q is expressed by

Q ¼ 1

10000ðN �MÞffiffiffiRp XR

i¼1

e2i

1þ log Aiþ RðAiÞ

Ai

� �2" #

ð9Þ

where N · M is the size of the image, R is the number ofregions, Ai and ei are the area in number of pixels andthe quadratic error of the color values of the ith region,respectively. Also, R(Ai) represents the number of regionsthat have an area equal to Ai. The smaller the value ofQ, the better the segmentation of the image. For imagesin grey levels, we have adapted the normalization term ofthe previous equation to obtain values within a similarrange to those obtained by Q in color images, and the def-inition of ei corresponds to the mean value of the grey levelof the ith region.

As case of study we use function Q both to objectivelyevaluate the quality of the algorithm, and to adjust thevalue of the parameters of the weight function of the edgesof the RAG. Weight parameters for each of the countersare introduced in Eq. (4). We have carried out an exhaus-tive study to determine the values of these parametersbased on the results obtained on applying the evaluationfunction Q to different test images. These images have beencreated artificially and present homogenous grey regions.Moreover, we can check the correct behavior of the algo-rithm because a manual segmentation of these test imagesis available. Along this section the partial segmentationsare performed using the SRG algorithm, except for themedical images (as we detail later). This way we show thatour proposal works using partial segmentations from dif-ferent techniques.

We show, as example, the behavior of the merging algo-rithm using different values for the image Test3. Weadjusted C1 = 1 as reference for all the cases. In Fig. 6the evaluation function Q compared to the number ofregions of the oversegmented image using four partial seg-mentations is shown. C2 is equal to 0.2, 0.4 and 0.6 inFig. 6(a)–(c), respectively. The best segmentation accordingto Q corresponds to its minimum, but it will be correct onlywhen this minimum coincides with the actual number ofregions of the image. In the figures a vertical line showsthe actual number of regions. In this case, 22 regions.The other parameters, C3 and C4, will take different valuesas shown in Fig. 6. In Table 1 the values of C1, C2, C3 andC4 used by the merging algorithm to obtain a correct seg-

0 5 10 15 20 25 30 35 40 45 500

200

400

600

800

1000

1200

1400

1600

0

200

400

600

800

1000

1200

1400

1600

Regions

Q

c3 = 0.6 c4 = 1.2c3 = 0.6 c4 = 1.6c3 = 0.6 c4 = 2

0 5 10 15 20 25 30 35 40 45 500

200

400

600

800

1000

1200

1400

1600

0

200

400

600

800

1000

1200

1400

1600

0

200

400

600

800

1000

1200

1400

1600

0

200

400

600

800

1000

1200

1400

1600

Regions

Q

c3 = 0.8 c4 = 1.2c3 = 0.8 c4 = 1.6c3 = 0.8 c4 = 2

0 5 10 15 20 25 30 35 40 45 50Regions

0 5 10 15 20 25 30 35 40 45 50Regions

0 5 10 15 20 25 30 35 40 45 50Regions

0 5 10 15 20 25 30 35 40 45 50Regions

Q

c3 = 1 c4 = 1.2c3 = 1 c4 = 1.6c3 = 1 c4 = 2

Q

c3 = 0.6 c4 = 1.2c3 = 0.6 c4 = 1.6c3 = 0.6 c4 = 2

Qc3 = 0.8 c4 = 1.2c3 = 0.8 c4 = 1.6c3 = 0.8 c4 = 2

Q

c3 = 1 c4 = 1.2c3 = 1 c4 = 1.6c3 = 1 c4 = 2

(a)

(b)

(c) 0 5 10 15 20 25 30 35 40 45 500

200

400

600

800

1000

1200

1400

1600

Regions

Q

c3 = 0.6 c4 = 1.2c3 = 0.6 c4 = 1.6c3 = 0.6 c4 = 2

0 5 10 15 20 25 30 35 40 45 500

200

400

600

800

1000

1200

1400

1600

Regions

Q

0

200

400

600

800

1000

1200

1400

1600

Q

c3 = 0.8 c4 = 1.2c3 = 0.8 c4 = 1.6c3 = 0.8 c4 = 2

0 5 10 15 20 25 30 35 40 45 50Regions

c3 = 1 c4 = 1.2c3 = 1 c4 = 1.6c3 = 1 c4 = 2

Fig. 6. Function Q compared to the number of regions of the oversegmented image using four partial segmentations for Test3. In this case, C1 = 1 and (a)C2 = 0.2, (b) C2 = 0.4 and (c) C2 = 0.6.

Table 1Values of the parameters that obtain a correct segmentation for the Test3

image

C1 C2 C3 C4

1 0.2 0.6 21 0.2 0.8 2

1 0.4 0.6 1.61 0.4 0.6 21 0.4 0.8 21 0.6 0.8 2


mentation are shown. These values fulfil the relationshipsof Eq. (5). After applying this study to a broad set of testimages we finally propose the following values: C1 = 1,C2 = 0.2, C3 = 0.8 and C4 = 2. We have to mention thatfor all the images we tested no correct segmentations areperformed using parameters with values that do not fulfilEq. (5).

Next, as example, we show the results obtained forthe images Test2 and Test3, that actually present 16 and22 regions, respectively. The segmentations obtained byapplying the algorithm together with the original imageare shown in Figs. 7 and 8. These figures also show the val-ues of function Q compared to the number of regions of theoversegmented images using 2, 3 and 4 different partial seg-mentations. From the behavior of the Q function, we caninfer that when the number of regions is equal to the num-ber that each image really has, Q presents its minimum.However, when only two partial segmentations are used,Q does not converge adequately. This is because thecreated oversegmented image do not correctly detect theregions of the images. Thus, with this example we showhow a sufficient number of partial segmentations warran-ties a correct final segmentation. Therefore, the selectionof an adequate number of partial segmentations is a funda-mental decision that affects the quality of the final result.

Fig. 7. Segmentation of the image Test2: (a) original image, (b) result ofthe segmentation and (c) Q compared to the number of regions. Fig. 8. Segmentation of the image Test3: (a) original image, (b) result of

the segmentation and (c) Q compared to the number of regions.


Fig. 8(b) shows the final segmentation obtained from thethree partial segmentations depicted in Fig. 2. Note that,using our proposal, a correct segmentation is obtainedfrom three partial segmentations that are not correct.

In order to illustrate the behavior of our proposal in amore precise way, in Fig. 9 we summarize the differentsteps of the global process for Test3 example. Initially, asinput, the algorithm uses three different partial segmen-tations of the image. None of them performs a correctsegmentation of Test3, but using shadowed zones, theydetect more details in the image than the original SRGalgorithm. Then, the combination of the partial segmenta-tions to obtain an oversegmented image is performed.Next, two regions are merged every iteration of the processas we have mentioned. Finally, the final global segmenta-tion of the image is obtained using function Q as stoppingcriterion.

Next, we are going to analyze the results obtained whenapplying the segmentation algorithm to real images of dif-ferent sizes and characteristics. Moreover, function Q isused to evaluate each of these segmentations, and the par-ticularities that their behavior present on these types ofimages are detailed. Shadowed zones, defined by differentthresholds, have been used in all the tests.

Fig. 10(a) shows Q versus the number of regions for theimage Tree. Note that a clear global minimum of Q does

not exist, so we cannot decide on which is the best segmen-tation with this criterion. The information that we canextract is that an interval of values of Q exists, shown inthe figure with an arrow, in which the most adequatesegmentations can be found.

A detailed view of this zone when N = 8 is shown inFig. 10(b). In this case the local minimums can be assessed,such as points a, b and c. The Tree original image and thesegmentations that produce these three points are shown inFig. 11(a)–(d), respectively. The first segmentation presentsjust 12 regions, and although it produces a low Q, it hasmerged too many regions, with the consequent loss ofdetails in the image. The next segmentation presents 22regions. Although the difference with other possible seg-mentations is small, the Q value in this case is the minimumof the whole function. The same conclusions can beobtained with the last segmentation, which has 44 regions,except that it has a higher Q value than the previous case.

Therefore, based on these results we conclude that forthe segmentation of real images, Q is very useful for deter-mining the set of the most adequate segmentations, but inmost of the cases it is not going to be sufficiently discrimi-nating to select just one.

Table 2 summarizes the results obtained on a set of realimages. N is the number of partial segmentations, Qmin is

Fig. 9. Example of the behavior of the algorithm using Test3 as test image: Creation of the oversegmented image and some iterations of the mergingalgorithm.

0 50 100 150 200 250 300 350 4000

500

1000

1500

2000

2500

Regions

Q

3 partial segm.4 partial segm.8 partial segm.

10 20 30 40 50 60 70 80 90 1000

100

200

300

400

500

600

700

800

900

1000

Regions

Q

a

b

c

(a) (b)

Fig. 10. Segmentation of the image Tree: (a) evaluation function Q and (b) detailed view for N = 8.


the minimum value of Q for this image, and R is the num-ber of regions that presents this segmentation. Moreover,

S5% is the number of possible segmentations with a Q valuethat differs in a maximum of 5% with respect to Qmin, and

Fig. 11. Segmentation of the image Tree: (a) original image, (b) 12regions, (c) 22 regions and (d) 44 regions.

Table 2Summary of results for real images

Images N Qmin R S5% Rmin�5% Rmax�5%

Tree 3 334.2 23 6 24 314 345.2 25 12 17 508 333.3 22 7 12 24

MIT 3 1446 50 18 21 584 1096.9 55 10 54 648 1092.5 64 8 65 72

House 3 125.8 35 8 30 384 144.5 24 6 25 308 126.9 42 17 30 49

Lena 4 289.2 73 17 52 1076 285.2 76 21 68 908 280.1 68 14 61 83

Peppers 3 283.5 56 8 57 644 294.1 81 25 82 1096 357.6 76 23 69 96

Man 3 604.5 166 105 101 2654 629.2 168 103 80 2666 561.1 120 49 97 146


Smin�5% and Smax�5% are the minimum and maximumnumber of regions of a segmentation considered on S5%,respectively.

We can note that the number of adequate segmentationsincreases as reflected by the value of S5%. In some cases,such as image Man, reaches more than 100. Function Q

does not provide enough discriminating information. Itindicates the difficulty of determining the best segmenta-tion. The number of regions that could be detected variesfor this image from 80 to 266 regions. However, for otherimages Q is quite significant. In any case, the fact that the

Fig. 12. Examples of final segmentation of real images: (a) MIT, (b) House, (c)SRG algorithm.

interval is wide, is not associated with the difficulty ofdetermining the best segmentation. Only the S5% segmenta-tions of this interval differ by less than 5% with respect toQmin.

In order to illustrate the results of the algorithm, Figs.12 and 13 show some examples of segmentations obtainedfor real images. Each of them corresponds to the valueQmin of the evaluation function. Note that using our pro-posal high quality segmentations are obtained. In Fig. 12the partial segmentations were obtained using the SRGalgorithm. Fig. 13 shows two medical images of the heart’sleft ventricle obtained using single photon emission com-puterized tomography (SPECT). The partial segmentationsof these images were obtained using an active contour

Lena and (d) Peppers. The partial segmentations were performed using the

Fig. 13. Examples of final segmentations of SPECT images. The partial segmentations were performed using an active contour model algorithm.

0

2

4

6

8

10

12

0 200 400 600 800 1000 1200 1400 1600Iterations

Tim

e(s

c)

0

20

40

60

80

100

120

140

160

0 1000 2000 3000 4000 5000Iterations

Tim

e(s

c)

(a) (b)

Fig. 14. Execution times of the merging algorithm for the images: (a) House and (b) Peppers.


model algorithm described by Weickert and Kuhne (2003).This algorithm was initialized using the method detailed in(Dosil et al.’s paper, 2005). The medical segmented imageswere validated visually by experts. In these examples, 8 dif-ferent partial segmentations to obtain each final segmentedimage were used.

Finally, Fig. 14 shows the execution times compared tothe number of iterations obtained on executing the mergingalgorithm on a Origin 200 system, for the images House

and Peppers, with sizes of 256 · 256 and 512 · 512 pixels,respectively.

6. Conclusions

In this paper, a new heuristic segmentation algorithm ispresented, based on the use of various segmentations of thesame image that are combined to create an oversegmentedimage. These segmentations can be performed using differ-ent techniques or even the same technique with different

initial conditions. A region-merging algorithm is thenapplied to this oversegmented image. The main idea ofour proposal is to obtain high quality segmentations froma set of low quality partial segmentations. A relevant aspectis that the information obtained from the partial segmenta-tions will, in fact, guide the merging process, in a such waythat the actual characteristics of each region or pixel arenot taken into account.

We introduce the concept of force of repulsion betweenneighboring pixels that indicates quantitatively theirtendency to form part of different regions. The force ofrepulsion considers several situations in which any twoneighboring pixels can be found in all the partial segmenta-tions that are used to create the oversegmented image.In this context, we introduce the concept of shadowedzone. The shadowed zones are groups of pixels that differin their intensity level a certain threshold from the regionin which they could be included. Introducing this conceptin the region-based algorithms, regions with low levels of


homogeneity are avoided improving the quality of thewhole process. Note that, given that the shadowed zonesare not treated by the algorithm, the information thatcan be extracted from these zones is minimum.

To deal with the problem of region merging we proposean iterative heuristic to merge the regions with the lowestforce of repulsion in each iteration. We use a weightedgraph and a heap for its management. As stopping crite-rion of the merging algorithm we can use different alterna-tives. As case of study we apply a function to evaluate thequality of the segmentation. This function has the advan-tage that it does not require any type of external parame-ters. Moreover, it has been used to adjust the parametersthat define the force of repulsion between neighboringpixels.

Once the algorithm has been tested on a group of artifi-cial test images, we apply the algorithm to a significantgroup of real images that have different sizes and character-istics, demonstrating the benefits of our proposal.

Acknowledgments

This study has been financed by the MCYT (Spain)through the research project TIC2001-3694-C02.

References

Beaulieu, J., Goldberg, M., 1989. Hierarchy in picture segmentation: Astepwise optimization approach. IEEE Trans. Pattern Anal. MachineIntell. 11 (2), 150–163.

Blake, A., Isard, M., 1998. Active Contours. Springer.Borsotti, M., Campadelli, P., Schettini, R., 1998. Quantitative evaluation

of color image segmentation results. Pattern Recognition Lett. 19,741–747.

Canny, J.F., 1986. A computational approach to edge-detection. IEEETrans. Pattern Anal. Machine Intell. 8, 679–698.

Chen, S., Lin, W., Chen, C., 1992. Split-and-merge image segmentationbased on localized feature analysis and statistical tests. CVGIP: Graph.Models Image Process. 53 (5), 457–475.

Dosil, R., Fdez-Vidal, X.R., Pardo, X.M., 2005. Decomposition of 3Dmedical images into visual patterns. IEEE Trans. Biomed. Eng. 52(12).

Fu, K., Mui, J., 1981. A survey on image segmentation. PatternRecognition 13 (1), 3–16.

Geman, D., Geman, S., 1984. Stochastic relaxation, Gibbs distributionand Bayesian restoration of images. IEEE Trans. Pattern Anal.Machine Intell. 6, 721–741.

Haralick, R., Shapiro, L., 1985. Survey, image segmentation techniques.Comput. Vision Graphics Image Process. 29 (1), 100–132.

Haris, K., Efstratiadis, N., Maglaveras, N., Katsaggelos, A.K., 1998.Hybrid image segmentation using watersheds and fast region merging.IEEE Trans. Image Process. 7 (12), 1684–1699.

Kanungo, T., Dom, B., Niblack, W., Steele, D., 1994. A fast algorithm forMDL-based multi-band image segmentation. Proc. IEEE Conf. onComputer Vision and Pattern Recognition, 609–616.

Kass, M., Witkin, A., Terzopoulos, D., 1988. Snakes: Active contourmodels. Internat. J. Computer Vision 1 (4), 321–331.

Knuth, D.E., 1973. The Art of Computer Programming: Sorting andSearching. Addison-Wesley.

Levine, M.D., Nazif, A.M., 1985. Dynamic measurement of computergenerated image segmentations. IEEE Trans. Pattern Anal. MachineIntell. PAMI 7 (2), 155–164.

Liu, J., Yang, Y., 1994. Multiresolution color image segmentation. IEEETrans. Pattern Anal. Machine Intell. 16 (7), 689–700.

Mehnert, A., Jackway, O., 1997. An improved seeded region growingalgorithm. Pattern Recognition Lett. 18 (10), 1065–1071.

Pal, N., Pal, S., 1993. A review on image segmentation techniques. PatternRecognition 26 (9), 1277–1299.

Rosenfeld, A., Kak, A., 1982. Digital Picture Processing. Academic Press,New York.

Sahoo, P., Soltani, S., Wong, A., Chen, Y., 1988. A survey of thresholdingtechniques. Comput. Vision Graphics Image Process. 41 (1), 233–260.

Shafarenko, L., Petrou, M., 1997. Automatic watershed segmentation ofrandomly textured color images. IEEE Trans. Pattern Anal. MachineIntell. 6 (11), 1530–1543.

Weickert, J., Kuhne, G., 2003. Fast methods for implicit active contourmodels. Geometric Level Set Methods in Imaging Vision andGraphics. Springer, New York, pp. 43–58.

Zhang, Y.J., 1996. A survey of evaluation methods for image segmenta-tion. Pattern Recognition 29 (8), 1335–1346.

Zhu, S.C., Yuille, A., 1996. Region competition: Unifying snakes, regiongrowing, and Bayes/MDL for multiband image segmentation. IEEETrans. Pattern Anal. Machine Intell. 18 (9), 884–900.

Image segmentation based on merging of sub-optimal segmentations

Documents

Transcript of Image segmentation based on merging of sub-optimal segmentations