Joint Optimization of Segmentation and Appearance...
Transcript of Joint Optimization of Segmentation and Appearance...
Joint Optimization of Segmentation and AppearanceModels
David Mandle, Sameep Tandon
April 29, 2013
David Mandle, Sameep Tandon (Stanford) April 29, 2013 1 / 19
Overview
1 Recap: Image Segmentation2 Optimization Strategy3 Experimental
David Mandle, Sameep Tandon (Stanford) April 29, 2013 2 / 19
Recap: Image Segmentation Problem
Segment an image into foreground and background
Figure: Left: Input image. Middle: Segmentation by EM (GrabCut). Right:Segmentation by the method covered today
David Mandle, Sameep Tandon (Stanford) April 29, 2013 3 / 19
Recap: Image Segmentation as Energy Optimization
Recall Grid Structured Markov Random Field:
Latent variables xi ∈ {0, 1} corresponding to foreground/backgroundObservations zi . Take to be RGB pixel valuesEdge potentials Φ(xi , zi ), Ψ(xi , xj)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 4 / 19
Recap: Image Segmentation as Energy Optimization
The Graphical Model encodes the following (unnormalized) probabilitydistribution:
David Mandle, Sameep Tandon (Stanford) April 29, 2013 5 / 19
Recap: Image Segmentation as Energy Optimization
Goal: find x to maximize P(x , z) (z is observed)
Taking logs:
E (x , z) =∑i
φ(xi , zi ) +∑i ,j
ψ(xi , xj)
Unary potential φ(xi , zi ) encodes how likely it is for a pixel or patch yito belong to segmentation xi .
Pairwise potential ψ(xi , xj) encodes neighborhood info aboutpixel/patch segmentation labels
David Mandle, Sameep Tandon (Stanford) April 29, 2013 6 / 19
Recap: Image Segmentation as Energy Optimization
Goal: find x to maximize P(x , z) (z is observed)
Taking logs:
E (x , z) =∑i
φ(xi , zi ) +∑i ,j
ψ(xi , xj)
Unary potential φ(xi , zi ) encodes how likely it is for a pixel or patch yito belong to segmentation xi .
Pairwise potential ψ(xi , xj) encodes neighborhood info aboutpixel/patch segmentation labels
David Mandle, Sameep Tandon (Stanford) April 29, 2013 6 / 19
Recap: Image Segmentation as Energy Optimization
Goal: find x to maximize P(x , z) (z is observed)
Taking logs:
E (x , z) =∑i
φ(xi , zi ) +∑i ,j
ψ(xi , xj)
Unary potential φ(xi , zi ) encodes how likely it is for a pixel or patch yito belong to segmentation xi .
Pairwise potential ψ(xi , xj) encodes neighborhood info aboutpixel/patch segmentation labels
David Mandle, Sameep Tandon (Stanford) April 29, 2013 6 / 19
Recap: Image Segmentation as Energy Optimization
Goal: find x to maximize P(x , z) (z is observed)
Taking logs:
E (x , z) =∑i
φ(xi , zi ) +∑i ,j
ψ(xi , xj)
Unary potential φ(xi , zi ) encodes how likely it is for a pixel or patch yito belong to segmentation xi .
Pairwise potential ψ(xi , xj) encodes neighborhood info aboutpixel/patch segmentation labels
David Mandle, Sameep Tandon (Stanford) April 29, 2013 6 / 19
Recap: GrabCut Model
Unary Potentials: log of Gaussian Mixture ModelI But to deal with tractability, we assign each xi to component ki
φ(xi , ki , θ|zi ) = − log π(xi , ki ) + logN(zi ;µ(ki ),Σ(ki ))
Pairwise Potentials:
ψ(xi , xj |zi , zj) = [xi 6= xj ] exp(−β−1‖zi − zj‖2)
where β = 2 · avg(‖zi − zj‖2)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 7 / 19
Recap: GrabCut Model
Unary Potentials: log of Gaussian Mixture ModelI But to deal with tractability, we assign each xi to component ki
φ(xi , ki , θ|zi ) = − log π(xi , ki ) + logN(zi ;µ(ki ),Σ(ki ))
Pairwise Potentials:
ψ(xi , xj |zi , zj) = [xi 6= xj ] exp(−β−1‖zi − zj‖2)
where β = 2 · avg(‖zi − zj‖2)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 7 / 19
Recap: GrabCut Optimization Strategy
GrabCut EM Algorithm
1 Initialize Mixture Models
2 Assign GMM components:
ki = arg minkφ(xi , ki , θ|zi )
3 Get GMM parameters:
θ = arg minθ
∑i
φ(xi , ki , θ|zi )
4 Perform segmentation using reduction to min-cut:
x = arg minx
E (x , z ; k , θ)
5 Iterate from step 2 until converged
David Mandle, Sameep Tandon (Stanford) April 29, 2013 8 / 19
Recap: GrabCut Optimization Strategy
GrabCut EM Algorithm
1 Initialize Mixture Models
2 Assign GMM components:
ki = arg minkφ(xi , ki , θ|zi )
3 Get GMM parameters:
θ = arg minθ
∑i
φ(xi , ki , θ|zi )
4 Perform segmentation using reduction to min-cut:
x = arg minx
E (x , z ; k , θ)
5 Iterate from step 2 until converged
David Mandle, Sameep Tandon (Stanford) April 29, 2013 8 / 19
Recap: GrabCut Optimization Strategy
GrabCut EM Algorithm
1 Initialize Mixture Models
2 Assign GMM components:
ki = arg minkφ(xi , ki , θ|zi )
3 Get GMM parameters:
θ = arg minθ
∑i
φ(xi , ki , θ|zi )
4 Perform segmentation using reduction to min-cut:
x = arg minx
E (x , z ; k , θ)
5 Iterate from step 2 until converged
David Mandle, Sameep Tandon (Stanford) April 29, 2013 8 / 19
Recap: GrabCut Optimization Strategy
GrabCut EM Algorithm
1 Initialize Mixture Models
2 Assign GMM components:
ki = arg minkφ(xi , ki , θ|zi )
3 Get GMM parameters:
θ = arg minθ
∑i
φ(xi , ki , θ|zi )
4 Perform segmentation using reduction to min-cut:
x = arg minx
E (x , z ; k , θ)
5 Iterate from step 2 until converged
David Mandle, Sameep Tandon (Stanford) April 29, 2013 8 / 19
Recap: GrabCut Optimization Strategy
GrabCut EM Algorithm
1 Initialize Mixture Models
2 Assign GMM components:
ki = arg minkφ(xi , ki , θ|zi )
3 Get GMM parameters:
θ = arg minθ
∑i
φ(xi , ki , θ|zi )
4 Perform segmentation using reduction to min-cut:
x = arg minx
E (x , z ; k , θ)
5 Iterate from step 2 until converged
David Mandle, Sameep Tandon (Stanford) April 29, 2013 8 / 19
New Model
Let’s consider a simpler model. This will be useful soon
Unary terms: Histograms
I K bins, bi is bin of pixel ziI θ0, θ1 ∈ [0, 1]K represent color models (distributions) over
foreground/backgroundI
φ(xi , bi , θ) = − log θxibi
Pairwise Potentialsψ(xi , xj) = wij |xi − xj |
We will define wij later; for now, consider pairwise equal to Grabcut
Total Energy:
E (x , θ0, θ1) =∑p∈V− logP(zp|θxp) +
∑(p,q)∈N
wpq|xp − xq|
P(zp|θxp) = θxpbp
David Mandle, Sameep Tandon (Stanford) April 29, 2013 9 / 19
New Model
Let’s consider a simpler model. This will be useful soon
Unary terms: HistogramsI K bins, bi is bin of pixel zi
I θ0, θ1 ∈ [0, 1]K represent color models (distributions) overforeground/background
I
φ(xi , bi , θ) = − log θxibi
Pairwise Potentialsψ(xi , xj) = wij |xi − xj |
We will define wij later; for now, consider pairwise equal to Grabcut
Total Energy:
E (x , θ0, θ1) =∑p∈V− logP(zp|θxp) +
∑(p,q)∈N
wpq|xp − xq|
P(zp|θxp) = θxpbp
David Mandle, Sameep Tandon (Stanford) April 29, 2013 9 / 19
New Model
Let’s consider a simpler model. This will be useful soon
Unary terms: HistogramsI K bins, bi is bin of pixel ziI θ0, θ1 ∈ [0, 1]K represent color models (distributions) over
foreground/background
I
φ(xi , bi , θ) = − log θxibi
Pairwise Potentialsψ(xi , xj) = wij |xi − xj |
We will define wij later; for now, consider pairwise equal to Grabcut
Total Energy:
E (x , θ0, θ1) =∑p∈V− logP(zp|θxp) +
∑(p,q)∈N
wpq|xp − xq|
P(zp|θxp) = θxpbp
David Mandle, Sameep Tandon (Stanford) April 29, 2013 9 / 19
New Model
Let’s consider a simpler model. This will be useful soon
Unary terms: HistogramsI K bins, bi is bin of pixel ziI θ0, θ1 ∈ [0, 1]K represent color models (distributions) over
foreground/backgroundI
φ(xi , bi , θ) = − log θxibi
Pairwise Potentialsψ(xi , xj) = wij |xi − xj |
We will define wij later; for now, consider pairwise equal to Grabcut
Total Energy:
E (x , θ0, θ1) =∑p∈V− logP(zp|θxp) +
∑(p,q)∈N
wpq|xp − xq|
P(zp|θxp) = θxpbp
David Mandle, Sameep Tandon (Stanford) April 29, 2013 9 / 19
New Model
Let’s consider a simpler model. This will be useful soon
Unary terms: HistogramsI K bins, bi is bin of pixel ziI θ0, θ1 ∈ [0, 1]K represent color models (distributions) over
foreground/backgroundI
φ(xi , bi , θ) = − log θxibi
Pairwise Potentialsψ(xi , xj) = wij |xi − xj |
We will define wij later; for now, consider pairwise equal to Grabcut
Total Energy:
E (x , θ0, θ1) =∑p∈V− logP(zp|θxp) +
∑(p,q)∈N
wpq|xp − xq|
P(zp|θxp) = θxpbp
David Mandle, Sameep Tandon (Stanford) April 29, 2013 9 / 19
New Model
Let’s consider a simpler model. This will be useful soon
Unary terms: HistogramsI K bins, bi is bin of pixel ziI θ0, θ1 ∈ [0, 1]K represent color models (distributions) over
foreground/backgroundI
φ(xi , bi , θ) = − log θxibi
Pairwise Potentialsψ(xi , xj) = wij |xi − xj |
We will define wij later; for now, consider pairwise equal to Grabcut
Total Energy:
E (x , θ0, θ1) =∑p∈V− logP(zp|θxp) +
∑(p,q)∈N
wpq|xp − xq|
P(zp|θxp) = θxpbp
David Mandle, Sameep Tandon (Stanford) April 29, 2013 9 / 19
EM under new model
1 Initialize histograms θ0, θ1.
2 Fix θ. Perform segmentation using reduction to min-cut:
x = arg minx
E (x , θ0, θ1)
3 Fix x. Compute θ0, θ1 (via standard parameter fitting).
4 Iterate from step 2 until converged
David Mandle, Sameep Tandon (Stanford) April 29, 2013 10 / 19
EM under new model
1 Initialize histograms θ0, θ1.
2 Fix θ. Perform segmentation using reduction to min-cut:
x = arg minx
E (x , θ0, θ1)
3 Fix x. Compute θ0, θ1 (via standard parameter fitting).
4 Iterate from step 2 until converged
David Mandle, Sameep Tandon (Stanford) April 29, 2013 10 / 19
EM under new model
1 Initialize histograms θ0, θ1.
2 Fix θ. Perform segmentation using reduction to min-cut:
x = arg minx
E (x , θ0, θ1)
3 Fix x. Compute θ0, θ1 (via standard parameter fitting).
4 Iterate from step 2 until converged
David Mandle, Sameep Tandon (Stanford) April 29, 2013 10 / 19
EM under new model
1 Initialize histograms θ0, θ1.
2 Fix θ. Perform segmentation using reduction to min-cut:
x = arg minx
E (x , θ0, θ1)
3 Fix x. Compute θ0, θ1 (via standard parameter fitting).
4 Iterate from step 2 until converged
David Mandle, Sameep Tandon (Stanford) April 29, 2013 10 / 19
Optimization
Goal: Minimize Energy
minxE (x)
E (x) =∑k
hk(n1k) +
∑(p,q)∈N
wpq|xp − xq|︸ ︷︷ ︸E1(x)
+ h(n1)︸ ︷︷ ︸E2(x)
where n1k =
∑p∈Vk
xp and n1 =∑
p∈V xp
This is hard!
But efficient strategies for optimizing E 1(x) and E 2(x) separately
David Mandle, Sameep Tandon (Stanford) April 29, 2013 11 / 19
Optimization
Goal: Minimize Energy
minxE (x)
E (x) =∑k
hk(n1k) +
∑(p,q)∈N
wpq|xp − xq|︸ ︷︷ ︸E1(x)
+ h(n1)︸ ︷︷ ︸E2(x)
where n1k =
∑p∈Vk
xp and n1 =∑
p∈V xp
This is hard!
But efficient strategies for optimizing E 1(x) and E 2(x) separately
David Mandle, Sameep Tandon (Stanford) April 29, 2013 11 / 19
Optimization
Goal: Minimize Energy
minxE (x)
E (x) =∑k
hk(n1k) +
∑(p,q)∈N
wpq|xp − xq|︸ ︷︷ ︸E1(x)
+ h(n1)︸ ︷︷ ︸E2(x)
where n1k =
∑p∈Vk
xp and n1 =∑
p∈V xp
This is hard!
But efficient strategies for optimizing E 1(x) and E 2(x) separately
David Mandle, Sameep Tandon (Stanford) April 29, 2013 11 / 19
Optimization via Dual Decomposition
Consider an optimization of the form
minx f1(x) + f2(x)
where optimizing f (x) = f1(x) + f2(x) is hard
But minx f1(x) and minx f2(x) are easy problems
Dual Decomposition idea: Optimize f1(x) and f2(x) separately andcombine in a principled way.
David Mandle, Sameep Tandon (Stanford) April 29, 2013 12 / 19
Optimization via Dual Decomposition
Consider an optimization of the form
minx f1(x) + f2(x)
where optimizing f (x) = f1(x) + f2(x) is hard
But minx f1(x) and minx f2(x) are easy problems
Dual Decomposition idea: Optimize f1(x) and f2(x) separately andcombine in a principled way.
David Mandle, Sameep Tandon (Stanford) April 29, 2013 12 / 19
Optimization via Dual Decomposition
Consider an optimization of the form
minx f1(x) + f2(x)
where optimizing f (x) = f1(x) + f2(x) is hard
But minx f1(x) and minx f2(x) are easy problems
Dual Decomposition idea: Optimize f1(x) and f2(x) separately andcombine in a principled way.
David Mandle, Sameep Tandon (Stanford) April 29, 2013 12 / 19
Optimization via Dual Decomposition
Consider an optimization of the form
minx f1(x) + f2(x)
where optimizing f (x) = f1(x) + f2(x) is hard
But minx f1(x) and minx f2(x) are easy problems
Dual Decomposition idea: Optimize f1(x) and f2(x) separately andcombine in a principled way.
David Mandle, Sameep Tandon (Stanford) April 29, 2013 12 / 19
Optimization via Dual Decomposition
Original Problem:minx f1(x) + f2(x)
Introduce local variables:
minx1,x2 f1(x1) + f2(x2)
s.t x1 = x2
Equivalent Problem:
minx1,x2 f1(x1) + f2(x2)
s.t x2 − x1 = 0
David Mandle, Sameep Tandon (Stanford) April 29, 2013 13 / 19
Optimization via Dual Decomposition
Original Problem:minx f1(x) + f2(x)
Introduce local variables:
minx1,x2 f1(x1) + f2(x2)
s.t x1 = x2
Equivalent Problem:
minx1,x2 f1(x1) + f2(x2)
s.t x2 − x1 = 0
David Mandle, Sameep Tandon (Stanford) April 29, 2013 13 / 19
Optimization via Dual Decomposition
Original Problem:minx f1(x) + f2(x)
Introduce local variables:
minx1,x2 f1(x1) + f2(x2)
s.t x1 = x2
Equivalent Problem:
minx1,x2 f1(x1) + f2(x2)
s.t x2 − x1 = 0
David Mandle, Sameep Tandon (Stanford) April 29, 2013 13 / 19
Optimization via Dual Decomposition
Primal Problem:
minx1,x2 f1(x1) + f2(x2)
s.t x2 − x1 = 0
Lagrangian Dual:
g(y) = minx1,x2
f1(x1) + f2(x2) + yT (x2 − x1)
Decompose Lagrangian Dual:
g(y) = (minx1
f1(x1)− yT x1) + (minx2
f2(x2) + yT x2)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 14 / 19
Optimization via Dual Decomposition
Primal Problem:
minx1,x2 f1(x1) + f2(x2)
s.t x2 − x1 = 0
Lagrangian Dual:
g(y) = minx1,x2
f1(x1) + f2(x2) + yT (x2 − x1)
Decompose Lagrangian Dual:
g(y) = (minx1
f1(x1)− yT x1) + (minx2
f2(x2) + yT x2)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 14 / 19
Optimization via Dual Decomposition
Primal Problem:
minx1,x2 f1(x1) + f2(x2)
s.t x2 − x1 = 0
Lagrangian Dual:
g(y) = minx1,x2
f1(x1) + f2(x2) + yT (x2 − x1)
Decompose Lagrangian Dual:
g(y) = (minx1
f1(x1)− yT x1) + (minx2
f2(x2) + yT x2)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 14 / 19
Optimization via Dual Decomposition
g(y) = (minx1
f1(x1)− yT x1)︸ ︷︷ ︸g1(y)
+ (minx2
f2(x2) + yT x2)︸ ︷︷ ︸g2(y)
For all y , g(y) is a lower bound on optimal of the primal problem.
Maximize g(y) w.r.t. y to get the tightest bound
Further, g(y) is concave in y . Many techniques for concavemaximization (subgradient ascent, etc).
Ideally have fast optimization strategies for g1(y) and g2(y)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 15 / 19
Optimization via Dual Decomposition
g(y) = (minx1
f1(x1)− yT x1)︸ ︷︷ ︸g1(y)
+ (minx2
f2(x2) + yT x2)︸ ︷︷ ︸g2(y)
For all y , g(y) is a lower bound on optimal of the primal problem.
Maximize g(y) w.r.t. y to get the tightest bound
Further, g(y) is concave in y . Many techniques for concavemaximization (subgradient ascent, etc).
Ideally have fast optimization strategies for g1(y) and g2(y)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 15 / 19
Optimization via Dual Decomposition
g(y) = (minx1
f1(x1)− yT x1)︸ ︷︷ ︸g1(y)
+ (minx2
f2(x2) + yT x2)︸ ︷︷ ︸g2(y)
For all y , g(y) is a lower bound on optimal of the primal problem.
Maximize g(y) w.r.t. y to get the tightest bound
Further, g(y) is concave in y . Many techniques for concavemaximization (subgradient ascent, etc).
Ideally have fast optimization strategies for g1(y) and g2(y)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 15 / 19
Optimization via Dual Decomposition
g(y) = (minx1
f1(x1)− yT x1)︸ ︷︷ ︸g1(y)
+ (minx2
f2(x2) + yT x2)︸ ︷︷ ︸g2(y)
For all y , g(y) is a lower bound on optimal of the primal problem.
Maximize g(y) w.r.t. y to get the tightest bound
Further, g(y) is concave in y . Many techniques for concavemaximization (subgradient ascent, etc).
Ideally have fast optimization strategies for g1(y) and g2(y)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 15 / 19
Optimization via Dual Decomposition
Back to our Segmentation problem
Energy
E (x) =∑k
hk(n1k) +
∑(p,q)∈N
wpq|xp − xq|+ h(n1)
where n1k =
∑p∈Vk
xp and n1 =∑
p∈V xp
Group terms
E (x) =∑k
hk(n1k) +
∑(p,q)∈N
wpq|xp − xq|︸ ︷︷ ︸E1(x)
+ h(n1)︸ ︷︷ ︸E2(x)
E (x) = E1(x) + E2(x)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 16 / 19
Optimization via Dual Decomposition
Back to our Segmentation problem
Energy
E (x) =∑k
hk(n1k) +
∑(p,q)∈N
wpq|xp − xq|+ h(n1)
where n1k =
∑p∈Vk
xp and n1 =∑
p∈V xp
Group terms
E (x) =∑k
hk(n1k) +
∑(p,q)∈N
wpq|xp − xq|︸ ︷︷ ︸E1(x)
+ h(n1)︸ ︷︷ ︸E2(x)
E (x) = E1(x) + E2(x)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 16 / 19
Optimization via Dual Decomposition
Back to our Segmentation problem
Energy
E (x) =∑k
hk(n1k) +
∑(p,q)∈N
wpq|xp − xq|+ h(n1)
where n1k =
∑p∈Vk
xp and n1 =∑
p∈V xp
Group terms
E (x) =∑k
hk(n1k) +
∑(p,q)∈N
wpq|xp − xq|︸ ︷︷ ︸E1(x)
+ h(n1)︸ ︷︷ ︸E2(x)
E (x) = E1(x) + E2(x)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 16 / 19
Optimization via Dual Decomposition
Dual Decomposition on E (x) = E 1(x) + E 2(x)
Φ(y) = minx1
(E 1(x1)− yT x1)︸ ︷︷ ︸Φ1(y)
+ minx2
(E 2(x2) + yT x2)︸ ︷︷ ︸Φ2(y)
Maximize Φ(y) w.r.t. y to get the tightest lower bound.
Φ1(y) can be computed efficiently via reduction to min s-t cut.
Φ2(y) via convex minimization (several strategies)
You could now use your favorite concave maximization strategy forΦ(y)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 17 / 19
Optimization via Dual Decomposition
Dual Decomposition on E (x) = E 1(x) + E 2(x)
Φ(y) = minx1
(E 1(x1)− yT x1)︸ ︷︷ ︸Φ1(y)
+ minx2
(E 2(x2) + yT x2)︸ ︷︷ ︸Φ2(y)
Maximize Φ(y) w.r.t. y to get the tightest lower bound.
Φ1(y) can be computed efficiently via reduction to min s-t cut.
Φ2(y) via convex minimization (several strategies)
You could now use your favorite concave maximization strategy forΦ(y)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 17 / 19
Optimization via Dual Decomposition
Dual Decomposition on E (x) = E 1(x) + E 2(x)
Φ(y) = minx1
(E 1(x1)− yT x1)︸ ︷︷ ︸Φ1(y)
+ minx2
(E 2(x2) + yT x2)︸ ︷︷ ︸Φ2(y)
Maximize Φ(y) w.r.t. y to get the tightest lower bound.
Φ1(y) can be computed efficiently via reduction to min s-t cut.
Φ2(y) via convex minimization (several strategies)
You could now use your favorite concave maximization strategy forΦ(y)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 17 / 19
Optimization via Dual Decomposition
Dual Decomposition on E (x) = E 1(x) + E 2(x)
Φ(y) = minx1
(E 1(x1)− yT x1)︸ ︷︷ ︸Φ1(y)
+ minx2
(E 2(x2) + yT x2)︸ ︷︷ ︸Φ2(y)
Maximize Φ(y) w.r.t. y to get the tightest lower bound.
Φ1(y) can be computed efficiently via reduction to min s-t cut.
Φ2(y) via convex minimization (several strategies)
You could now use your favorite concave maximization strategy forΦ(y)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 17 / 19
Optimization via Dual Decomposition
Dual Decomposition on E (x) = E 1(x) + E 2(x)
Φ(y) = minx1
(E 1(x1)− yT x1)︸ ︷︷ ︸Φ1(y)
+ minx2
(E 2(x2) + yT x2)︸ ︷︷ ︸Φ2(y)
Maximize Φ(y) w.r.t. y to get the tightest lower bound.
Φ1(y) can be computed efficiently via reduction to min s-t cut.
Φ2(y) via convex minimization (several strategies)
You could now use your favorite concave maximization strategy forΦ(y)
David Mandle, Sameep Tandon (Stanford) April 29, 2013 17 / 19
Optimization via Dual Decomposition
Turns out can maximize Φ(y) over y in polynomial time using aparametric max-flow technique.
Sketch:
I Theorem: Given Φ1(y) and Φ2(y) as described, optimal y has formy = s1
I Implication: maximize Φ(s1) over all possible sI Φ1(s1) is piecewise-linear concave
F |V | breakpoints computed by parametric max-flowF Parametric max-flow also returns between 2 and |V |+ 1 solutions
segmentations x per breakpoint
I Φ2(s1) is piecewise-linear concaveF |V | breakpoints can be enumerated from h(·)
I Φ(s1) is piecewise-linear concave with 2|V | breakpoints, so findingmax over Φ(1s) is easy – it’s one of the breakpoints.
I Given the breakpoint, return the segmentation x with minimum energyfrom set given by PMF.
David Mandle, Sameep Tandon (Stanford) April 29, 2013 18 / 19
Optimization via Dual Decomposition
Turns out can maximize Φ(y) over y in polynomial time using aparametric max-flow technique.
Sketch:
I Theorem: Given Φ1(y) and Φ2(y) as described, optimal y has formy = s1
I Implication: maximize Φ(s1) over all possible sI Φ1(s1) is piecewise-linear concave
F |V | breakpoints computed by parametric max-flowF Parametric max-flow also returns between 2 and |V |+ 1 solutions
segmentations x per breakpoint
I Φ2(s1) is piecewise-linear concaveF |V | breakpoints can be enumerated from h(·)
I Φ(s1) is piecewise-linear concave with 2|V | breakpoints, so findingmax over Φ(1s) is easy – it’s one of the breakpoints.
I Given the breakpoint, return the segmentation x with minimum energyfrom set given by PMF.
David Mandle, Sameep Tandon (Stanford) April 29, 2013 18 / 19
Optimization via Dual Decomposition
Turns out can maximize Φ(y) over y in polynomial time using aparametric max-flow technique.
Sketch:I Theorem: Given Φ1(y) and Φ2(y) as described, optimal y has form
y = s1
I Implication: maximize Φ(s1) over all possible sI Φ1(s1) is piecewise-linear concave
F |V | breakpoints computed by parametric max-flowF Parametric max-flow also returns between 2 and |V |+ 1 solutions
segmentations x per breakpoint
I Φ2(s1) is piecewise-linear concaveF |V | breakpoints can be enumerated from h(·)
I Φ(s1) is piecewise-linear concave with 2|V | breakpoints, so findingmax over Φ(1s) is easy – it’s one of the breakpoints.
I Given the breakpoint, return the segmentation x with minimum energyfrom set given by PMF.
David Mandle, Sameep Tandon (Stanford) April 29, 2013 18 / 19
Optimization via Dual Decomposition
Turns out can maximize Φ(y) over y in polynomial time using aparametric max-flow technique.
Sketch:I Theorem: Given Φ1(y) and Φ2(y) as described, optimal y has form
y = s1I Implication: maximize Φ(s1) over all possible s
I Φ1(s1) is piecewise-linear concaveF |V | breakpoints computed by parametric max-flowF Parametric max-flow also returns between 2 and |V |+ 1 solutions
segmentations x per breakpoint
I Φ2(s1) is piecewise-linear concaveF |V | breakpoints can be enumerated from h(·)
I Φ(s1) is piecewise-linear concave with 2|V | breakpoints, so findingmax over Φ(1s) is easy – it’s one of the breakpoints.
I Given the breakpoint, return the segmentation x with minimum energyfrom set given by PMF.
David Mandle, Sameep Tandon (Stanford) April 29, 2013 18 / 19
Optimization via Dual Decomposition
Turns out can maximize Φ(y) over y in polynomial time using aparametric max-flow technique.
Sketch:I Theorem: Given Φ1(y) and Φ2(y) as described, optimal y has form
y = s1I Implication: maximize Φ(s1) over all possible sI Φ1(s1) is piecewise-linear concave
F |V | breakpoints computed by parametric max-flowF Parametric max-flow also returns between 2 and |V |+ 1 solutions
segmentations x per breakpoint
I Φ2(s1) is piecewise-linear concaveF |V | breakpoints can be enumerated from h(·)
I Φ(s1) is piecewise-linear concave with 2|V | breakpoints, so findingmax over Φ(1s) is easy – it’s one of the breakpoints.
I Given the breakpoint, return the segmentation x with minimum energyfrom set given by PMF.
David Mandle, Sameep Tandon (Stanford) April 29, 2013 18 / 19
Optimization via Dual Decomposition
Turns out can maximize Φ(y) over y in polynomial time using aparametric max-flow technique.
Sketch:I Theorem: Given Φ1(y) and Φ2(y) as described, optimal y has form
y = s1I Implication: maximize Φ(s1) over all possible sI Φ1(s1) is piecewise-linear concave
F |V | breakpoints computed by parametric max-flowF Parametric max-flow also returns between 2 and |V |+ 1 solutions
segmentations x per breakpoint
I Φ2(s1) is piecewise-linear concaveF |V | breakpoints can be enumerated from h(·)
I Φ(s1) is piecewise-linear concave with 2|V | breakpoints, so findingmax over Φ(1s) is easy – it’s one of the breakpoints.
I Given the breakpoint, return the segmentation x with minimum energyfrom set given by PMF.
David Mandle, Sameep Tandon (Stanford) April 29, 2013 18 / 19
Optimization via Dual Decomposition
Turns out can maximize Φ(y) over y in polynomial time using aparametric max-flow technique.
Sketch:I Theorem: Given Φ1(y) and Φ2(y) as described, optimal y has form
y = s1I Implication: maximize Φ(s1) over all possible sI Φ1(s1) is piecewise-linear concave
F |V | breakpoints computed by parametric max-flowF Parametric max-flow also returns between 2 and |V |+ 1 solutions
segmentations x per breakpoint
I Φ2(s1) is piecewise-linear concaveF |V | breakpoints can be enumerated from h(·)
I Φ(s1) is piecewise-linear concave with 2|V | breakpoints, so findingmax over Φ(1s) is easy – it’s one of the breakpoints.
I Given the breakpoint, return the segmentation x with minimum energyfrom set given by PMF.
David Mandle, Sameep Tandon (Stanford) April 29, 2013 18 / 19
Optimization via Dual Decomposition
Turns out can maximize Φ(y) over y in polynomial time using aparametric max-flow technique.
Sketch:I Theorem: Given Φ1(y) and Φ2(y) as described, optimal y has form
y = s1I Implication: maximize Φ(s1) over all possible sI Φ1(s1) is piecewise-linear concave
F |V | breakpoints computed by parametric max-flowF Parametric max-flow also returns between 2 and |V |+ 1 solutions
segmentations x per breakpoint
I Φ2(s1) is piecewise-linear concaveF |V | breakpoints can be enumerated from h(·)
I Φ(s1) is piecewise-linear concave with 2|V | breakpoints, so findingmax over Φ(1s) is easy – it’s one of the breakpoints.
I Given the breakpoint, return the segmentation x with minimum energyfrom set given by PMF.
David Mandle, Sameep Tandon (Stanford) April 29, 2013 18 / 19
References
MRF Review Slides:http://vision.stanford.edu/teaching/cs231b spring1213/slides/segmentation.pdfDual Decomposition Slides: Petter Strandmark and Fredrik Kahl. Paralleland Distributed Graph Cuts by Dual Decomposition.http://www.robots.ox.ac.uk/vgg/rg/slides/parallelgraphcuts.pdf
David Mandle, Sameep Tandon (Stanford) April 29, 2013 19 / 19