Comprehensive Cross-Hierarchy Cluster Agreement Evaluation

David M. Johnson, Caiming Xiong, Jing Gao and Jason J. CorsoState University of New York at Buffalo

Department of Computer Science and Engineering338 Davis Hall

Buffalo, NY, 14260-2500{davidjoh,cxiong,jing,jcorso}@buffalo.edu

Abstract

Hierarchical clustering represents a family of widely usedclustering approaches that can organize objects into a hierar-chy based on the similarity in objects’ feature values. One sig-nificant obstacle facing hierarchical clustering research todayis the lack of general and robust evaluation methods. Existingworks rely on a range of evaluation techniques including bothinternal (no ground-truth is considered in evaluation) andexternal measures (results are compared to ground-truth se-mantic structure). The existing internal techniques may havestrong hierarchical validity, but the available external mea-sures were not developed specifically for hierarchies. Thislack of specificity prevents them from comparing hierarchystructures in a holistic, principled way. To address this prob-lem, we propose the Hierarchy Agreement Index, a novel hi-erarchy comparison algorithm that holistically compares twohierarchical clustering results (e.g. ground-truth and automat-ically learned hierarchies) and rates their structural agreementon a 0-1 scale. We compare the proposed evaluation methodwith a baseline approach (based on computing F-Score resultsbetween individual nodes in the two hierarchies) on a set ofunsupervised and semi-supervised hierarchical clustering re-sults, and observe that the proposed Hierarchy Agreement In-dex provides more intuitive and reasonable evaluation of thelearned hierarchies.

Introduction and Related WorkResearch into flat clustering methods benefits greatly fromthe availability of powerful cluster evaluation tools, rangingfrom the original Rand Index (Rand 1971) to more mod-ern methods such as V-Measure (Rosenberg and Hirschberg2007), that allow researches to effectively judge their clus-tering results against some ground-truth objective, and thuscompare the relative performance of different methods.

Unfortunately, hierarchical clustering does not enjoy sucha wealth of viable, established evaluation techniques. Whilethere do exist well-grounded internal (i.e. with no groundtruth information) measures of hierarchy quality (?), re-searchers interested on measuring the external, semanticmeaning of a hierarchy are forced to resort to a wide array ofquestionable and limited methods. Some, for instance, sim-ply test their methods on non-hierarchical data, and cut their

Copyright c© 2013, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

hierarchies to produce a flat segmentation for evaluation (?;?).

Currently, the most common external evaluation tech-nique is a method we refer to as Cluster F-Measure (CFM).It was originally proposed in (?) and works (on either flator hierarchical clusterings) by matching each ground-truthcluster/hierarchy node to the “best” test cluster/hierarchynode (where match quality is determined by F (c1, c2) =2·P ·RP+R : the F-score for cluster c2 on cluster c1). The overallscore, then, is just the cluster-size-weighted average of thebest scores. This method is applicable to hierarchies, but notspecific to them, and when applied to a hierarchical cluster-ing solution completely ignores the actual structure of thehierarchy, treating each node as an unrelated flat cluster. Wewould argue that a strong tool for evaluating hierarchicalclustering solutions must take account of the hierarchical re-lationships between nodes and elements, and that the CFMapproach is thus inadequate.

We instead propose a novel evaluation technique that di-rectly encodes and compares the entire structure of two hier-archies. We refer to our method, which can be thought of asan extension of the classical Rand Index to the hierarchicalcase, as the Hierarchy Agreement Index (HAI).

Hierarchy Agreement IndexIn both the Rand Index and our proposed HAI, the total com-parison score is computed via:

S(C1, C2) = 1− 1N2

N∑i=1

N∑j=1

|f(xi, xj , C1)−f(xi, xj , C2)| ,

(1)where C1 and C2 are the two clustering solutions being com-pared, N is the number of elements being clustered and f isa function that extracts some value in the range [0,1] describ-ing the relatedness of two points under the given clusteringscheme.

For the Rand Index, f(xi, xj , C) is an indicator function,returning 1 if xi and xj are are in the same cluster under Cand 0 otherwise. For hierarchy evaluation, we instead needan f that can return a range of values representing an arbi-trary number of different degrees of relatedness. Moreover,f must be general to a wide range of different hierarchystructures. Most notably, it must yield meaningful values for

Figure 1: Results of hierarchical clustering with varyingnumbers of constraints on an example dataset we created.The data consists of 2500 randomly generated RGB colorvalues, with a “ground truth” hierarchy constructed by run-ning UPGMA (?) on the data in Lab space. We then rana semi-supervised hierarchical clustering algorithm on thedata (in HSV space), with varying numbers of triplet con-straints drawn from the ground truth. The resulting hierar-chies are visualized above. (view in color)

both binary and non-binary hierarchy trees, as well as bothdeep hierarchies (such as those produced by agglomerativeclustering algorithms) and shallow hierarchies (such as mosthuman-generated semantic hierarchies).

To satisfy these requirements, we define the hierarchy dis-tance dH(a, b) between two elements a and b in cluster hi-erarchy H. Let na,b be the smallest node in the hierarchycontaining both a and b, and let size(n) = |nD|

N (i.e. theproportion of the total element set found in node n). Now letdH(a, b) = size(na,b), or 0 if na,b is a leaf node (becausetwo elements that lie in the same leaf node are maximallyclose under that hierarchy).

Using hierarchy distance, our cluster agreement measurebecomes:

HAI(H1,H2) = 1− 1N2

N∑i=1

N∑j=1

|dH1(xi, xj)−dH2(xi, xj)|

(2)The distance dH is independent of the specific structure of

the hierarchy because it is unaffected by the number of in-tervening nodes between xi, xj and nxi,xj . Additionally, bydefining the hierarchy distance between two elements withinthe same leaf node as 0, we can meaningfully represent dis-tance even in shallow hierarchies. While shallow hierarchy

Constraints HAI CFM0 0.701 0.4891298 0.705 0.4727142 0.749 0.44214026 0.856 0.45223442 0.923 0.51244791 0.991 0.784

Table 1: HAI (this paper) and CFM evaluation againstground truth on synthetic color data.

trees may contain very large leaf nodes, distance measureswithin these trees will not be distorted because node-pairswithin those leaf nodes will still be considered maximallyclose.

Thus, using equation 2, we can compare the relatedness ofeach point pair in the dataset under two input hierarchies andaggregate the results to achieve a single unified measure ofhierarchy correspondence. Like the Rand Index, this mea-sure will yield a correspondence score of 1 only for caseswhere the two cluster solutions are functionally identical(“functionally” because, again, two hierarchy trees, such asa binary and non-binary tree, may encode the same informa-tion despite having somewhat different tree structures).

ExperimentsTo validate our measure, we performed semi-supervised hi-erarchical clustering, with varying numbers of constraints,on a set of easily-visualized synthetic color data (see Fig-ure 1). We then used both CFM and HAI to measure theagreement between each resulting hierarchy and the groundtruth. The details of the particular semi-supervised hierar-chical clustering method are beyond the scope of this shortpaper, but in essence it uses triplet constraints drawn fromthe ground truth hierarchy to iteratively perform flat semi-supervised clustering on the data, ultimately yielding a divi-sive hierarchy.

Figure 1 shows that as we add constraints to the semi-supervised method the resulting cluster hierarchy increas-ingly appears to resemble the ground truth hierarchy. TheHAI results in Table 1 coincide well with this intuition,showing first minor gains, then a significant improvementby the 14000 constraint level. By contrast, the CFM mea-sure actually worsens initially, and fails to detect any im-provement in the hierarchy until the 23000 constraint level.

Conclusion and Future WorkEffective cluster hierarchy analysis is an important problem,and we believe HAI can fill a needed niche as an evalua-tion tool for hierarchical clustering algorithms. In additionto exploring variants on the HAI algorithm—an adjusted-for-chance version, for instance—we plan to perform a morerigorous evaluation of the measure, with more theoretically-grounded test data/hierarchies. However, we believe this ini-tial experiment offers some useful validation for the method.

ReferencesRand, W. 1971. Objective criteria for the evaluation of clus-tering methods. JASA.Rosenberg, A., and Hirschberg, J. 2007. V-measure: A con-ditional entropy-based external cluster evaluation measure.In EMNLP-CoNLL.