of 24

• date post

18-Mar-2020
• Category

Documents

• view

0

0

Embed Size (px)

Transcript of Normalized Cuts and Image Segmentation 1 Normalized Cuts and Image Segmentation Jianbo Shi, Upenn...

• 1

Normalized Cuts and Image Segmentation

Jianbo Shi, Upenn Jitendra Malik, Berkeley

10/7/2003 2

Perceptual Grouping and Organization (Wertheimer)

Similarity Proximity Common fate Good continuation Past experience

= Visual grouping

• 2

10/7/2003 3

History

10/7/2003 4

How to Partition?

There is no single answer Prior World Knowledge:

Low Level Coherence of Brightness Color Texture Motion

Mid or High Level Symmetries of Objects

• 3

10/7/2003 5

How to Partition? (Cont’d)

inherently hierarchical TREE STRUCTURE instead of flat partitioning OBJECTIVE:

low level cues for hierarchical partitions. High level knowledge for further partitioning.

10/7/2003 6

Tools

G=(V,E) An edge is formed between every pair (Complete graph) w(i,j) function of similarity. Partition V into disjoint sets V1, V2,…, Vm

Similarity within sets maximum. Similarity between sets minimum.

• 4

10/7/2003 7

Graph Vs. Image

What is the best criterion i.e. the weighting function How to compute efficiently

The graph is huge

10/7/2003 8

Definitions

Optimal bipartition is minimum cut Minimum cut

computed efficiently partitions out small chunks

• 5

10/7/2003 9

Similarity decreases as the distance increases

10/7/2003 10

New Definition: Normalized Cut

Normalize the cut. Still a disassociation (dissimilarity) measure.

Smaller the better.

• 6

10/7/2003 11

New Definition: Normalized Association

Total association (similarity) within groups. The bigger the better.

assoc(a,a): total connectivity within A.

10/7/2003 12

Normalized Cut and Normalized Association Are Related

• 7

10/7/2003 13

Formulation for Normal Cut

10/7/2003 14

System Linearization

• 8

10/7/2003 15

Setting up the matrices

d1 0 0 0 0 0 …… 0 0 d2 0 0 0 0 …… 0 D-W=

D=0 0 d3 0 0 0 …… 0 ………………………. 0 0 0 0 0 0 …… dN

0 w12 w13 …. w1N W= w21 0 w23 …. w2N

…………………….. wN1 wN2 ….. …. 0

dN…-wN3-wN2-wN1

……………

-w3N…d3-w32-w31

-w2N…-w23d2-w21

-w1N…-w13-w12d1

10/7/2003 16

Rayleigh Quotient

Generalized eigenvalue system Minimized if y can be real.

Convert generalized eigensystem to a standard eigensystem:

Where

• 9

10/7/2003 17

Results

D-W is positive semidefinite eigenvectors are perpendicular.

Second smallest eigenvector minimizes the normalized cut.

Rayleigh quotient

Constraints (to be justified later): y can take on either 1 or –b

10/7/2003 18

Problem

First constraint not satisfied The solution gives real valued eigenvectors.

Which pixels are above threshold, which ones are not?

• 10

10/7/2003 19

Algorithm

Set up G=(V,E) All the points in the image as nodes Complete graph.

Weights proportional to the similarity between two pixels.

Similarity measure is yet to be decided.

10/7/2003 20

Algorithm (cont’d)

Solve (D-W)x=λDx for smallest eigenvectors Method is yet to be decided.

Second smallest eigenvector bipartitions the graph.

Must decide the break point. Decide if current partition is good, if not recursively subdivide.

• 11

10/7/2003 21

Example: Brightness Image

Define the weighting function To build matrices D and W:

Sigma: 10 to 20 percent of the total range of the distance function.

10/7/2003 22

Example (Cont’d)

Solve for Transform into a standard eigenvalue problem

Normally takes O(n^3) operations Use Lanczos method to reduce computation.

• 12

10/7/2003 23

Lanczos Method

Graphs are locally connected, thus sparse Eigensystems are therefore sparse

Only few smallest eigenvectors needed Precision requirement for eigenvectors is loose.

10/7/2003 24

Dividing Point for Eigenvector

Ideal case: second smallest eigenvector is integer. But in reality it takes real values

Use the median, or Use 0, or Use the value that makes Ncut(A,B) smallest.

• 13

10/7/2003 25

Stability Criterion

Sometimes an eigenvector takes on continues values. Not suitable for partitioning purpose

Hard to find a cut point Similar Ncut values.

Compute the histogram ratio between the minimum and maximum values in the bins.

10/7/2003 26

Drawbacks of 2-Way Cut

Stability criterion is good but it gets rid of subsequent eigenvectors

Recursive 2-way cut is inefficient uses only the second smallest eigenvector

• 14

10/7/2003 27

Simultaneous K-Way Cut with Multiple Eigenvectors

k-means is used for oversegmentation. Second step can vary

Greedy Pruning Global Recursive Cut

10/7/2003 28

Greedy Pruning

Iteratively merge two segments until k segments are left. Each iteration choose the pair when merged minimizes the k-way Ncut criterion:

• 15

10/7/2003 29

Global Recursive Cut

Build a condensed graph Each node of the graph corresponds to one initial segment Weights Defined as the total connection from one initial partition to another Use exhaustive search or generalized eigenvalue system for solution.

10/7/2003 30

Experiments

Brightness Color Texture Motion Following equation always holds; it is F(i) that changes according to the application

• 16

10/7/2003 31

Similarity * Spatial Proximity

F(i)= 1, when segmenting point sets I(i), the intensity value for segmenting scalar images [v,v.s.sin(h), v.s.cos(h)](i) for color images [|I*f1|, … , |I*fn|](i), for texture.

The weight is always zero if proximity criterion is not met, i.e. The pixels are more than r pixels apart.

10/7/2003 32

MOTION

Spatiotemporal neighborhood

Dm: motion distance

• 17

10/7/2003 33

Experimental Results

10/7/2003 34

Experimental Results

• 18

10/7/2003 35

Experimental Results

10/7/2003 36

• 19

10/7/2003 37

10/7/2003 38

Motion

• 20

10/7/2003 39

10/7/2003 40

• 21

10/7/2003 41

10/7/2003 42

Linearize the System

Xi > 0 if node i belongs to A, Xi

• 22

10/7/2003 43

Linearize the System (2)

The equation linearizes to:

D is a diagonal NxN matrix with d on its diagonal, W is the NxN weight matrix where w(i,j) is the weight of the edge between node i and j. 1 is a vector of all ones.

10/7/2003 44

Linearize the System (3)

• 23

10/7/2003 45

Linearize the System (4)

10/7/2003 46

Linearize the System (5)

• 24

10/7/2003 47