1

Distributed Cosegmentation via Submodular Optimization on

Anisotropic Diffusion

Gunhee Kim1 Eric P. Xing1 Li Fei-Fei2 Takeo Kanade1

1: School of Computer Science, Carnegie Mellon University2: Computer Science Department, Stanford University

November 9, 2011

2

• Problem Statement• Submodular Optimization on Diffusion • Applications

Diversity Ranking Single Image Segmentation Cosegmentation

• Experiments• Conclusion

Outline

3

• Problem Statement • Submodular Optimization on Diffusion • Applications

Diversity Ranking Single Image Segmentation Cosegmentation

• Experiments• Conclusion

Outline

4

Image Cosegmentation

Remove the ambiguity of what should be segmented out?

Jointly segment M images into K regions !

• Rother et al. 2006• Hochbaum and Singh, 2009• Joulin et al, 2010• Batra et al, 2010• Mukherjee et al, 2011• Vincente et al, 2010, 2011

(M = 3, K = 2)

5

Why is Cosegmentation Interesting?

Wide potential in Web applications

Photo-taking patterns of general users

My son joined baseball club.

I saw dolphins in aquarium.

6

Our Approach

Major challenges for web photos

Jointly segment M images into K regions

Work Model / Algorithm M K

Ours[J10]

[M11][H09][R06][B10][V10]

Anisotropic Diffusion/ SubmodularityDiscriminative ClusteringMRF+ Rank-1 global / Iterative opt.MRF+Reward global / Graph CutsMRF+L1 global / Trust Region GCBoykov-Jolly / Graph CutsBoykov-Jolly / Dual Decomposition

103

30 3022

502

Any222222

• [R06] Rother et al. 2006• [H09] Hochbaum and Singh, 2009• [B10] Batra et al, 2010

• [J10] Joulin et al, 2010• [V10] Vincente et al, 2010• [M11] Mukherjee et al, 2011

(1) Large-scale (2) Highly-variable

7

Contributions

A New optimization framework

Work Model / Algorithm M K

Ours[J10]

[M11][H09][R06][B10][V10]

Anisotropic Diffusion/ SubmodularityDiscriminative ClusteringMRF+ Rank-1 global / Iterative opt.MRF+Reward global / Graph CutsMRF+L1 global / Trust Region GCBoykov-Jolly / Graph CutsBoykov-Jolly / Dual Decomposition

103

30 3022

502

Any222222

• Constant-factor approximation of optimal• Easily parallelizable• Automatic selection of K • Robust against wrong K

• [R06] Rother et al. 2006• [H09] Hochbaum and Singh, 2009• [B10] Batra et al, 2010

• [J10] Joulin et al, 2010• [V10] Vincente et al, 2010• [M11] Mukherjee et al, 2011

8

• Problem Statement• Submodular Optimization on Diffusion• Applications

Diversity Ranking Single Image Segmentation Cosegmentation

• Experiments• Conclusion

Outline

9

Diffusion

Diffusion in physics

Spread of particles (or energy) through random motionfrom high concentration to low concentration

Examples• Electric current• Heat diffusion

[Wikipedia]

Heat Equation (Partial Differential Equation)

Temperature Diffusivity (conductance) tensor

10

Optimization

Maximize the sum of temperature of the system

max

s.t

K heat sources

Environment temperature

Maximize the sum of temperature

11

Correspondences

Temperature maximization and Image Segmentation

max

s.t

Heat Diffusion Points Temperature Heat sources Conductance

Image Segmentation

Pixels Segmentation confidence

Segment centers Similarity btw features of pixels

Image Segmentation

Select K pixels as segment centers, to maximize sum of segmentation confidence of every pixel.

12

Optimization

How can we solve this?

max

s.t

[Theorem] (Neuhauser, Wolsey, Fisher 1978)Let u be , nondecreasing, and submodular.Then, the greedy algorithm finds a set such that

0.632.

13

Submodularity on Anisotropic Diffusion

[Theorem] Suppose that system is under linear anisotropic diffusion

(T1)

Let be temperature at time t point x when heat sources are attached to Then, the following holds for

(T2) is nondecreasing

(T3) is submodular

(Proof)

14

Submodularity on Anisotropic Diffusion

[Theorem] Suppose that system is under linear anisotropic diffusion

(T1)

Let be temperature at time t point x when heat sources are attached to Then, the following holds for

(T2) is nondecreasing

(T3) is submodular

(Proof)

x x

(Diminishing Return)

Induction on distance

15

Greedy Algorithm

Sketch of the greedy algorithm

max

s.t

Find the point with maximum marginal gain in every round.

1. 2. Iterate until

2.1.

2.2. Marginal gain

16

• Problem Statement• Submodular Optimization on Diffusion• Applications

Diversity Ranking Single Image Segmentation Cosegmentation

• Experiments• Conclusion

Outline

17

Diversity Ranking

Ranking items according to both centrality and diversity

Items

Rankingvalues

A BC

Centrality only: A > B > CCentrality + Diversity: A > C > B

18

Optimization for Diversity Ranking

Simplification

max

s.t

(2) Steady-state

(4) Every v is connected to the ground with z

(1) System is a graph

(3) Diffusivity is defined by Gaussian similarity

19

Optimization for Diversity Ranking

Simplification

max

s.t

(2) Steady-state

(4) Every v is connected to the ground with z

(1) System is a graph

max

s.t

(3) Diffusivity is defined by Gaussian similarity

20

Examples of Diversity Ranking

Input data

(1) vertices

(2) features

(3) Conductance

21

Examples of Diversity Ranking

Input data 2nd item

3rd item Clustering

Marginal gain

1st item

22

• Problem Statement• Submodular Optimization on Diffusion• Applications

Diversity Ranking Single Image Segmentation Cosegmentation

• Experiments• Conclusion

Outline

23

Segmenting a Single Image

Input image 1. Superpixels (SP)

G = (V, E, W) G = (V, E, W)

2. Connect adjacent SPs

g(v) = ColorTexture

Construct image graph G = (V, E, W)

G = (V, E, W)

Optimization formulation is similar to that of diversity ranking

3. Features on SP 4. Conductance

24

Basic Behavior of Our Segmentation

Greedily select the largest and most coherent regions !

Input image K=2: sky K=3: tree K=4: wall

K=5: roof K=6: window K=7: building K=8: trash can

Source code is available !

Automatic selection of K

25

• Problem Statement• Submodular Optimization on Diffusion• Applications

Diversity Ranking Single Image Segmentation Cosegmentation

• Experiments• Conclusion

Outline

26

Cosegmentation

Segment selection should be coupled!

Single image segmentation

Objective 1: Segment should be large and coherent.

Objective 2: Segment should be similar to its corresponding ones in other images

+

Cosegmentation

27

Cosegmentation

Control source temperatures

A

B

A

B

Cosegmentation

A is better than Bto maximize the temperature

28

An Toy Example of Cosegmentation

Input images Likelihood Cosegmentation Segments

MSRC cow images (M=3, K=4)

Source code is available !

29

• Problem Statement• Submodular Optimization on Diffusion• Applications

Diversity Ranking Single Image Segmentation Cosegmentation

• Experiments• Conclusion

Outline

30

Two Experiments

Figure-ground cosegmentation with a pair of images

Scalable cosegmentation

• Goal: Compare with other state-of-the-art techniques

• Dataset: MSRC

ex. cat

• Goal: Feasibility for Web photos

• Dataset: ImageNet

ex. green lizard

31

Exp1. Figure-Ground Cosegmentation

Segmentation accuracies for 100 random pairs of MSRC

[6, 7] Use their implementation without modification

[6] ICCV 2009[7] CVPR 2010

32

Cosegmentation on MSRC

Cosegmentation Examples (K=8)

Ours (K = 8)(1) Multiple instances(2) Robust against wrong choice of K

Normalized cuts (K = 8)

33

• Problem Statement• Submodular Optimization on Diffusion• Applications

Diversity Ranking Single Image Segmentation Cosegmentation

• Experiments• Conclusion

Outline

34

Conclusion

Prove the temperature in anisotropic diffusion is submodular.

(1) A large-scale edge-preserving image smoothing(2) Layered motion segmentation

Diversity ranking Cosegmentation Single-image segmentation

Source code is available !

Next step

What’s done

smoothing Optical flow

35

Conclusion

What’s done

Cosegmentation for Web photos was proposed• Arbitrary K and a larger M by order of magnitude • Easily Parallelizable• Automatic selection of K • Robust against wrong K

(Ours) (Ncuts)

36

Thank you !Stop by our poster at 80!