Normalized Cuts and Image Segmentation 1 Normalized Cuts and Image Segmentation Jianbo Shi, Upenn...

download Normalized Cuts and Image Segmentation 1 Normalized Cuts and Image Segmentation Jianbo Shi, Upenn Jitendra

of 24

  • date post

    18-Mar-2020
  • Category

    Documents

  • view

    0
  • download

    0

Embed Size (px)

Transcript of Normalized Cuts and Image Segmentation 1 Normalized Cuts and Image Segmentation Jianbo Shi, Upenn...

  • 1

    Normalized Cuts and Image Segmentation

    Jianbo Shi, Upenn Jitendra Malik, Berkeley

    10/7/2003 2

    Perceptual Grouping and Organization (Wertheimer)

    Similarity Proximity Common fate Good continuation Past experience

    = Visual grouping

  • 2

    10/7/2003 3

    History

    10/7/2003 4

    How to Partition?

    There is no single answer Prior World Knowledge:

    Low Level Coherence of Brightness Color Texture Motion

    Mid or High Level Symmetries of Objects

  • 3

    10/7/2003 5

    How to Partition? (Cont’d)

    inherently hierarchical TREE STRUCTURE instead of flat partitioning OBJECTIVE:

    low level cues for hierarchical partitions. High level knowledge for further partitioning.

    10/7/2003 6

    Tools

    G=(V,E) An edge is formed between every pair (Complete graph) w(i,j) function of similarity. Partition V into disjoint sets V1, V2,…, Vm

    Similarity within sets maximum. Similarity between sets minimum.

  • 4

    10/7/2003 7

    Graph Vs. Image

    What is the best criterion i.e. the weighting function How to compute efficiently

    The graph is huge

    10/7/2003 8

    Definitions

    Optimal bipartition is minimum cut Minimum cut

    computed efficiently partitions out small chunks

  • 5

    10/7/2003 9

    Example of a Bad Partition

    Similarity decreases as the distance increases

    10/7/2003 10

    New Definition: Normalized Cut

    Normalize the cut. Still a disassociation (dissimilarity) measure.

    Smaller the better.

  • 6

    10/7/2003 11

    New Definition: Normalized Association

    Total association (similarity) within groups. The bigger the better.

    assoc(a,a): total connectivity within A.

    10/7/2003 12

    Normalized Cut and Normalized Association Are Related

  • 7

    10/7/2003 13

    Formulation for Normal Cut

    10/7/2003 14

    System Linearization

  • 8

    10/7/2003 15

    Setting up the matrices

    d1 0 0 0 0 0 …… 0 0 d2 0 0 0 0 …… 0 D-W=

    D=0 0 d3 0 0 0 …… 0 ………………………. 0 0 0 0 0 0 …… dN

    0 w12 w13 …. w1N W= w21 0 w23 …. w2N

    …………………….. wN1 wN2 ….. …. 0

    dN…-wN3-wN2-wN1

    ……………

    -w3N…d3-w32-w31

    -w2N…-w23d2-w21

    -w1N…-w13-w12d1

    10/7/2003 16

    Rayleigh Quotient

    Generalized eigenvalue system Minimized if y can be real.

    Convert generalized eigensystem to a standard eigensystem:

    Where

  • 9

    10/7/2003 17

    Results

    D-W is positive semidefinite eigenvectors are perpendicular.

    Second smallest eigenvector minimizes the normalized cut.

    Rayleigh quotient

    Constraints (to be justified later): y can take on either 1 or –b

    10/7/2003 18

    Problem

    First constraint not satisfied The solution gives real valued eigenvectors.

    Which pixels are above threshold, which ones are not?

  • 10

    10/7/2003 19

    Algorithm

    Set up G=(V,E) All the points in the image as nodes Complete graph.

    Weights proportional to the similarity between two pixels.

    Similarity measure is yet to be decided.

    10/7/2003 20

    Algorithm (cont’d)

    Solve (D-W)x=λDx for smallest eigenvectors Method is yet to be decided.

    Second smallest eigenvector bipartitions the graph.

    Must decide the break point. Decide if current partition is good, if not recursively subdivide.

  • 11

    10/7/2003 21

    Example: Brightness Image

    Define the weighting function To build matrices D and W:

    Sigma: 10 to 20 percent of the total range of the distance function.

    10/7/2003 22

    Example (Cont’d)

    Solve for Transform into a standard eigenvalue problem

    Normally takes O(n^3) operations Use Lanczos method to reduce computation.

  • 12

    10/7/2003 23

    Lanczos Method

    Graphs are locally connected, thus sparse Eigensystems are therefore sparse

    Only few smallest eigenvectors needed Precision requirement for eigenvectors is loose.

    10/7/2003 24

    Dividing Point for Eigenvector

    Ideal case: second smallest eigenvector is integer. But in reality it takes real values

    Use the median, or Use 0, or Use the value that makes Ncut(A,B) smallest.

  • 13

    10/7/2003 25

    Stability Criterion

    Sometimes an eigenvector takes on continues values. Not suitable for partitioning purpose

    Hard to find a cut point Similar Ncut values.

    Compute the histogram ratio between the minimum and maximum values in the bins.

    10/7/2003 26

    Drawbacks of 2-Way Cut

    Stability criterion is good but it gets rid of subsequent eigenvectors

    Recursive 2-way cut is inefficient uses only the second smallest eigenvector

  • 14

    10/7/2003 27

    Simultaneous K-Way Cut with Multiple Eigenvectors

    k-means is used for oversegmentation. Second step can vary

    Greedy Pruning Global Recursive Cut

    10/7/2003 28

    Greedy Pruning

    Iteratively merge two segments until k segments are left. Each iteration choose the pair when merged minimizes the k-way Ncut criterion:

  • 15

    10/7/2003 29

    Global Recursive Cut

    Build a condensed graph Each node of the graph corresponds to one initial segment Weights Defined as the total connection from one initial partition to another Use exhaustive search or generalized eigenvalue system for solution.

    10/7/2003 30

    Experiments

    Brightness Color Texture Motion Following equation always holds; it is F(i) that changes according to the application

  • 16

    10/7/2003 31

    Similarity * Spatial Proximity

    F(i)= 1, when segmenting point sets I(i), the intensity value for segmenting scalar images [v,v.s.sin(h), v.s.cos(h)](i) for color images [|I*f1|, … , |I*fn|](i), for texture.

    The weight is always zero if proximity criterion is not met, i.e. The pixels are more than r pixels apart.

    10/7/2003 32

    MOTION

    Spatiotemporal neighborhood

    Dm: motion distance

  • 17

    10/7/2003 33

    Experimental Results

    10/7/2003 34

    Experimental Results

  • 18

    10/7/2003 35

    Experimental Results

    10/7/2003 36

  • 19

    10/7/2003 37

    10/7/2003 38

    Motion

  • 20

    10/7/2003 39

    10/7/2003 40

  • 21

    10/7/2003 41

    10/7/2003 42

    Linearize the System

    Xi > 0 if node i belongs to A, Xi

  • 22

    10/7/2003 43

    Linearize the System (2)

    The equation linearizes to:

    D is a diagonal NxN matrix with d on its diagonal, W is the NxN weight matrix where w(i,j) is the weight of the edge between node i and j. 1 is a vector of all ones.

    10/7/2003 44

    Linearize the System (3)

  • 23

    10/7/2003 45

    Linearize the System (4)

    10/7/2003 46

    Linearize the System (5)

  • 24

    10/7/2003 47