Iterative Optimization and Simplification of Hierarchical Clusterings

Doug FisherDepartment of Computer Science,Vanderbilt University

Journal of Artificial Intelligence Research,4 (1996) 147-179

Presented by: Biyu Liang

2

Outline

Introduction Generating Initial Hierarchical Clustering Iterative Optimization Methods and

Comparison Simplification of Hierarchical Clustering Conclusion

3

Introduction

Clustering is a process of unsupervised learning, which groups objects into clusters.

Major Clustering Methods Partitioning Hierarchical Density-based Grid-based Model-based

4

Introduction (Continued)

Clustering systems differ in• objective function • control strategy

Usually a search strategy cannot be both computationally inexpensive and give any guarantee about the quality.

5

Introduction (Continued) This paper discusses the use of iterative

optimization and simplification to construct clusters that satisfy both conditions:

High quality Computationally inexpensive

The suggested method involves 3 steps: Constructing a initial clustering inexpensively Iterative optimization to improve the

clustering Retrospective simplification of the clustering

6

Outline

Introduction Generating Initial Hierarchical Clustering Iterative Optimization Methods and

Experiments Simplification of Hierarchical Clustering Conclusion

7

Category Utility

CU(CK) = P(Ck)ij[P(Ai = Vij |CK)2 -P(Ai = Vij)2]

PU({C1, C2, … CN}) = k CU(CK)/N

Where an observation is a vector of Vij along attributes(or variables) Ai

This measure rewards clusters Ck, that increases the predictability of Vij within Ck (i.e. P(Ai=Vij|Ck)) relative to their predictability in the population as a whole (i.e. P(Ai = Vij))

8

P(Color=gre|C1)

9

Hierarchical Sorting

Given an observation and current partition, evaluate the quality of the clusterings that result from Placing the observation in each of the existing

clusters Creating a new cluster that only covers the new

observation Select the option that yields the highest

quality score (PU)

10

11

Outline

Introduction Generating Initial Hierarchical Clustering Iterative Optimization Methods and

Comparison Simplification of Hierarchical Clustering Conclusion

12

Iterative Optimization Methods Reorder-resort (Cluster/2): seed selection,

reordering, and re-clustering. Iterative redistribution of single observation:

moving single observation one by one. Iterative hierarchical redistribution: moving

clusters together with its sub-tree.

13

Reorder-resort (k-mean)

k random seeds are selected, and k clusters are growing around these attractors

the centroids of the clusters are picked as new seeds, new clusters are growing

The process iterates until there is no further improvement in the quality of generated clustering

14

Reorder-resort (k-mean) con’t Ordering data to make consecutive

observations dissimilar leads to good clusterings

Extracting biased “dissimilarity” ordering from the hierarchical clustering

Initial sorting, extraction dissimilarity ordering, re-clustering

15

Iterative Redistribution of Single Observations

Moves single observations from cluster to cluster

A cluster contains only one observation is removed and its single observation is resorted

Iterate until two consecutive iterations yield the same clustering

16

The ISODATA algorithm determines a target cluster for each observation but does not move the cluster until targets for all observations have been determined

A sequential version that moves each observation as its target is identified through sorting

Single Observation Redistribution Variations

17

Iterative Hierarchical Redistribution Takes large steps in the search for a

better clustering Remove and resorts sub-tree instead of

single observation Requires update variable value counts

of ancestor clusters and host cluster

18

Scheme

Given an existing hierarchical clustering, a recursive loop examines sibling clusters in the hierarchy in a depth first fashion.

An inner, iterative loop reclassifies each sibling based on the objective function. And repeats until two consecutive iterations lead to the same set of siblings.

19

(Continued)

The recursive loop then turns its attention to the children of each of these remaining siblings.

Finally the leaves will be reached and resorted.

The recursive loop will be applied several times until there are no changes that occur from one pass to the next.

20

21

Main findings from the experiments Hierarchical redistribution achieves the highest

mean PU scores in most cases Reordering and re-clustering comes closest to

hierarchical redistribution’s performance in all cases Single-observation redistribution modestly improves

an initial sort, and is substantially worse than the other two optimization methods

22

Outline

Introduction Generating Initial Hierarchical Clustering Iterative Optimization Methods and

Comparison Simplification of Hierarchical Clustering Conclusion

23

Simplifying Hierarchical Clustering Simplify hierarchical clustering and minimize

classification cost Minimize Error Rate Validation set to identify the frontier of

clusters for prediction of each variable Node lies below the frontier of every variable

would be pruned

24

Validation For each variable, Ai, the objects from the validation

set are each classified through the hierarchical clustering with the value of variable Ai “masked” for purposes of classification.

At each cluster encountered during classification, prediction correct if the observation’s value for Ai is equal to the most frequent value for Ai at the cluster.

A Count of all correct predictions for each variable at a cluster is maintained.

A preferred frontier for each variable is identified that maximizes the number of correct counts for the variable.

25

26

Concluding Remarks

There are three phases in searching the space of hierarchical clusterings: Inexpensive generation of an initial clustering Iterative optimization for clusterings Retrospective simplification of generated

clusterings Experiments found that the new method,

hierarchical redistribution optimization works well

Thanks! Question?

28

Final Exam Questions

The main idea in this paper is to construct clusterings which satisfy two conditions.

Name the conditions (p.5) name the two steps to satisfy the conditions

Discribe the three iterative methods for clustering optimization (p.12-20)

The cluster is better when the relative CU score is a) big, b) small, c) equal to 0 (p.7)

Which sorting method is better? a) random sorting, b) similarity sorting (p.14)