1

Normalized Cuts and Image Segmentation

Jianbo Shi, UpennJitendra Malik, Berkeley

10/7/2003 2

Perceptual Grouping and Organization (Wertheimer)

SimilarityProximityCommon fateGood continuationPast experience

=Visual grouping

2

10/7/2003 3

History

10/7/2003 4

How to Partition?

There is no single answerPrior World Knowledge:

Low LevelCoherence of BrightnessColorTextureMotion

Mid or High LevelSymmetries of Objects

3

10/7/2003 5

How to Partition? (Cont’d)

inherently hierarchicalTREE STRUCTURE instead of flat partitioningOBJECTIVE:

low level cues for hierarchical partitions. High level knowledge for further partitioning.

10/7/2003 6

Tools

G=(V,E)An edge is formed between every pair (Complete graph)w(i,j) function of similarity.Partition V into disjoint sets V1, V2,…, Vm

Similarity within sets maximum.Similarity between sets minimum.

4

10/7/2003 7

Graph Vs. Image

What is the best criterion i.e. the weighting functionHow to compute efficiently

The graph is huge

10/7/2003 8

Definitions

Optimal bipartition is minimum cutMinimum cut

computed efficientlypartitions out small chunks

5

10/7/2003 9

Example of a Bad Partition

Similarity decreases as the distance increases

10/7/2003 10

New Definition: Normalized Cut

Normalize the cut.Still a disassociation (dissimilarity) measure.

Smaller the better.

6

10/7/2003 11

New Definition: Normalized Association

Total association (similarity) within groups.The bigger the better.

assoc(a,a): total connectivity within A.

10/7/2003 12

Normalized Cut and Normalized Association Are Related

7

10/7/2003 13

Formulation for Normal Cut

10/7/2003 14

System Linearization

8

10/7/2003 15

Setting up the matrices

d1 0 0 0 0 0 …… 0 0 d2 0 0 0 0 …… 0 D-W=

D=0 0 d3 0 0 0 …… 0……………………….0 0 0 0 0 0 …… dN

0 w12 w13 …. w1NW= w21 0 w23 …. w2N

……………………..wN1 wN2 ….. …. 0

dN…-wN3-wN2-wN1

……………

-w3N…d3-w32-w31

-w2N…-w23d2-w21

-w1N…-w13-w12d1

10/7/2003 16

Rayleigh Quotient

Generalized eigenvalue systemMinimized if y can be real.

Convert generalized eigensystem to a standard eigensystem:

Where

9

10/7/2003 17

Results

D-W is positive semidefiniteeigenvectors are perpendicular.

Second smallest eigenvector minimizes the normalized cut.

Rayleigh quotient

Constraints (to be justified later):y can take on either 1 or –b

10/7/2003 18

Problem

First constraint not satisfiedThe solution gives real valued eigenvectors.

Which pixels are above threshold, which ones are not?

10

10/7/2003 19

Algorithm

Set up G=(V,E)All the points in the image as nodesComplete graph.

Weights proportional to the similarity between two pixels.

Similarity measure is yet to be decided.

10/7/2003 20

Algorithm (cont’d)

Solve (D-W)x=λDx for smallest eigenvectorsMethod is yet to be decided.

Second smallest eigenvector bipartitions the graph.

Must decide the break point.Decide if current partition is good, if not recursively subdivide.

11

10/7/2003 21

Example: Brightness Image

Define the weighting function To build matrices D and W:

Sigma: 10 to 20 percent of the total range of the distance function.

10/7/2003 22

Example (Cont’d)

Solve for Transform into a standard eigenvalue problem

Normally takes O(n^3) operationsUse Lanczos method to reduce computation.

12

10/7/2003 23

Lanczos Method

Graphs are locally connected, thus sparseEigensystems are therefore sparse

Only few smallest eigenvectors neededPrecision requirement for eigenvectors is loose.

10/7/2003 24

Dividing Point for Eigenvector

Ideal case: second smallest eigenvector is integer. But in reality it takes real values

Use the median, orUse 0, orUse the value that makes Ncut(A,B) smallest.

13

10/7/2003 25

Stability Criterion

Sometimes an eigenvector takes on continues values.Not suitable for partitioning purpose

Hard to find a cut point Similar Ncut values.

Compute the histogram ratio between the minimum and maximum values in the bins.

10/7/2003 26

Drawbacks of 2-Way Cut

Stability criterion is good but it gets rid of subsequent eigenvectors

Recursive 2-way cut is inefficientuses only the second smallest eigenvector

14

10/7/2003 27

Simultaneous K-Way Cut with Multiple Eigenvectors

k-means is used for oversegmentation.Second step can vary

Greedy PruningGlobal Recursive Cut

10/7/2003 28

Greedy Pruning

Iteratively merge two segments until k segments are left.Each iteration choose the pair when merged minimizes the k-way Ncut criterion:

15

10/7/2003 29

Global Recursive Cut

Build a condensed graphEach node of the graph corresponds to one initial segmentWeights Defined as the total connection from one initial partition to anotherUse exhaustive search or generalized eigenvalue system for solution.

10/7/2003 30

Experiments

BrightnessColorTextureMotionFollowing equation always holds; it is F(i) that changes according to the application

16

10/7/2003 31

Similarity * Spatial Proximity

F(i)=1, when segmenting point setsI(i), the intensity value for segmenting scalar images[v,v.s.sin(h), v.s.cos(h)](i) for color images[|I*f1|, … , |I*fn|](i), for texture.

The weight is always zero if proximity criterion is not met, i.e. The pixels are more than r pixels apart.

10/7/2003 32

MOTION

Spatiotemporal neighborhood

Dm: motion distance

17

10/7/2003 33

Experimental Results

10/7/2003 34

Experimental Results

18

10/7/2003 35

Experimental Results

10/7/2003 36

19

10/7/2003 37

10/7/2003 38

Motion

20

10/7/2003 39

10/7/2003 40

21

10/7/2003 41

10/7/2003 42

Linearize the System

Xi > 0 if node i belongs to A, Xi<0 otherwise.

is the connection from node i to all nodes.

22

10/7/2003 43

Linearize the System (2)

The equation linearizes to:

D is a diagonal NxN matrix with d on its diagonal, W is the NxN weight matrix where w(i,j) is the weight of the edge between node i and j.1 is a vector of all ones.

10/7/2003 44

Linearize the System (3)

23

10/7/2003 45

Linearize the System (4)

10/7/2003 46

Linearize the System (5)

24

10/7/2003 47