“GrabCut” — Interactive Foreground Extraction Using Iterated Graph Cuts
Graph Cuts vs. Level Sets - University of Western Ontario · · 2006-05-29Graph Cuts vs. Level...
Transcript of Graph Cuts vs. Level Sets - University of Western Ontario · · 2006-05-29Graph Cuts vs. Level...
ECCV 2006 tutorial on
Graph Cuts vs. Level Sets
part IV
Global vs. local optimisation algorithms
Vladimir Kolmogorov
University College London
Yuri Boykov
University of Western Ontario
Daniel Cremers
University of Bonn
Global vs. local minima
(Geodesic active contours)
• Geodesic active contours
– Variational approach (e.g. level sets)
• Gradient descent in the space of contours
• Local minimum
• Non-convex formulation?
– Graph cuts (e.g. geo-cuts)
• Same problem, global minimum
• Convex formulation?
[Anonymous attendee of CVPR’05]:
How is it possible?
Global vs. local minima
(Geodesic active contours)
• Geodesic active contours
– Variational approach (e.g. level sets)
• Gradient descent in the space of contours
• Local minimum
• Non-convex formulation?
– Graph cuts (e.g. geo-cuts)
• Same problem, global minimum
• Convex formulation?
[Anonymous attendee of CVPR’05]:
How is it possible?
Graph cuts
• Function E(x) of discrete variables: convexity not defined
• Extend the space of solutions: xp∈{0,1} ⇒ xp∈[0,1]
– Allow fractional segmentations
• Extend energy: linear programming (LP) relaxation
– Now convex problem!
Energy function
with discrete variables
E
submodular function ⇒ integer solution
LP relaxation
E tight
Graph cuts
• Function E(x) of discrete variables: convexity not defined
• Extend the space of solutions: xp∈{0,1} ⇒ xp∈[0,1]
– Allow fractional segmentations
• Extend energy: linear programming (LP) relaxation
– Now convex problem!
Energy function
with discrete variables
E Enot tight
non-submodular function ⇒ fractional solution (in general)
LP relaxation
Solving LP relaxation
• Too large for general purpose LP solvers (e.g. interior point methods)
• Solve dual problem instead of primal:
– Formulate lower bound on the energy
– Maximize this bound
– When done, solves primal problem (LP relaxation)
• Two different ways to formulate lower bound
– Part A: Via posiforms => maxflow algorithm (for binary variables)
– Part B: Via convex combination of trees => tree-reweighted message passing
Lower bound on
the energy function
E
Energy function
with discrete variables
E E
LP relaxation
Notation and Preliminaries
Energy function
∑∑ ++=qp
qppq
p
ppconst xxxE,
),()()|( θθθθx
p
qunary terms
(data)pairwise terms
(coherence)
- xp are discrete variables (for example, xp∈{0,1})
- θp(•) are unary potentials
- θpq(•,•) are pairwise potentials
LP relaxation
• [Schlesinger’76,Koster et al.’98,Chekuri et al.’00,Wainwright et al.’03]
• Introduce indicator variables xp;i , xpq;ij
=
=
→+
∑
∑
∑∑
i
ip
j
ipijpq
jiqp
ijpqpq
ip
ipp
x
xx
xjixi
1
min),()(
;
;;
,,,
;
,
; θθ
xpq;ij∈{0,1}relaxation
xpq;ij∈[0,1]
Energy function - visualisation
0
4
0
1
3
02
5
node p edge (p,q) node q
label 0
label 1
)0(pθ
)1,0(pqθ
0
constθ
∑∑ ++=qp
qppq
p
ppconst xxxE,
),()()|( θθθθx
0
4
0
1
3
02
5
node p edge (p,q) node q
label 0
label 1
Energy function - visualisation
∑∑ ++=qp
qppq
p
ppconst xxxE,
),()()|( θθθθx
0
vector of
all parameters=θ
0 0 4
4
1 12
5
0
-1
-1
0 + 1
Reparameterisation
Reparameterisation
0 0 3
4
1 02
5
• Definition. is a reparameterisation of
if they define the same energy:
xxx any for )|()|( θθ EE =′
( )θθ ≡′θ ′ θ
4 -1
1 -1 0 +1
• Maxflow, BP and TRW perform reparameterisations
1
Part A: Lower bound
via posiforms
(⇒⇒⇒⇒ maxflow algorithm)
non-negative
constθ - lower bound on the energy:
xx ∀≥ constE θθ )|(
maximize
Lower bound via posiforms[Hammer, Hansen, Simeone’84]
∑∑ ++=qp
qppq
p
ppconst xxxE,
),()()|( θθθθx
• Posiform maximisation: algorithm?
• Binary variables, submodular functions
– Reduction to maxflow
– Global minimum of the energy
• Binary variables, non-submodular functions
– Reduction to maxflow
• More complicated graph
– Part of optimal solution
Outline of part A
Posiform maximisation
Binary variables,
submodular functions
• Definition: E is submodular if every pairwise term satisfies
• Can be converted to “canonical form”:
Submodularity and canonical form
)0,1()1,0()1,1()0,0( pqpqpqpq θθθθ +≤+
2
1 2 3 4
10
0 05
zero
cost
Overview of min cut/max flow
Min Cut problem
source
sink
2 1
1
2
3
4
5
Directed weighted graph
Min Cut problem
sink
2 1
1
2
3
4
5
S = {source, node 1}
T = {sink, node 2, node 3}
Cut:
source
Min Cut problem
sink
2 1
1
2
3
4
5
S = {source, node 1}
T = {sink, node 2, node 3}
Cut:
• Task: Compute cut with minimum cost
Cost(S,T) = 1 + 1 = 2
source
sink
2 1
1
2
3
4
5
source
Maxflow algorithm
value(flow)=0
Maxflow algorithm
sink
2 1
1
2
3
4
5
value(flow)=0
source
Maxflow algorithm
sink
1 1
1
2
3
4
5
value(flow)=0
source
Maxflow algorithm
sink
1 1
0
2
3
4
5
value(flow)=0
source
Maxflow algorithm
sink
1 1
0
2
3
4
4
value(flow)=1
source
Maxflow algorithm
sink
1 1
0
3
3
4
4
value(flow)=1
source
Maxflow algorithm
sink
1 1
0
3
3
4
4
value(flow)=1
source
Maxflow algorithm
sink
1 1
0
3
3
4
4
value(flow)=1
source
Maxflow algorithm
sink
1 0
0
3
4
3
3
value(flow)=2
source
Maxflow algorithm
sink
1 0
0
3
4
3
3
value(flow)=2
source
value(flow)=2
sink
1 0
0
3
4
3
3
source
Maxflow algorithm
Posiform maximisation
Binary variables,
non-submodular functions
Reduction to maxflow
2
1 2 3 4
10
0 05
Maxflow algorithm
and reparameterisation
2
1 2 3 4
10
0 05
sink
5
source
Maxflow algorithm
and reparameterisation
2
1 2 3 4
10
0 05
sink
2
5
source
Maxflow algorithm
and reparameterisation
2
1 2 3 4
10
0 05
sink
2
5
source
Maxflow algorithm
and reparameterisation
2
1 2 3 4
10
0 05
sink
2
1
5
source
Maxflow algorithm
and reparameterisation
2
1 2 3 4
10
0 05
sink
2
1
5
source
Maxflow algorithm
and reparameterisation
2
1 2 3 4
10
0 05
sink
2
1
2
5
source
Maxflow algorithm
and reparameterisation
2
1 2 3 4
10
0 05
sink
2
1
2
5
source
Maxflow algorithm
and reparameterisation
2
1 2 3 4
10
0 05
sink
2
1
2
5
source
Maxflow algorithm
and reparameterisation
2
1 2 3 4
10
0 05
sink
2
1
2
5
source
Maxflow algorithm
and reparameterisation
2
1 2 3 4
10
0 05
sink
2
1
2
3
4
5
source
Maxflow algorithm
and reparameterisation
2
1 2 3 4
10
0 05
sink
2
1
2
3
4
5
source
Maxflow algorithm
and reparameterisation
2
1 2 3 4
10
0 05
sink
2 1
1
2
3
4
5
source
0
Maxflow algorithm
and reparameterisation
2
1 2 3 4
10
0 05
sink
2 1
1
2
3
4
5
source
value(flow)=0
0
Maxflow algorithm
and reparameterisation
sink
2 1
1
2
3
4
5
value(flow)=0
2
1 2 3 4
10
0 05
0
source
Maxflow algorithm
and reparameterisation
sink
1 1
0
3
3
4
4
value(flow)=1
1
0 3 3 4
10
0 04
1
source
Maxflow algorithm
and reparameterisation
sink
1 1
0
3
3
4
4
value(flow)=1
1
0 3 3 4
10
0 04
1
source
Maxflow algorithm
and reparameterisation
sink
1 0
0
3
4
3
3
value(flow)=2
1
0 3 4 3
00
0 03
2
source
Maxflow algorithm
and reparameterisation
sink
1 0
0
3
4
3
3
value(flow)=2
1
0 3 4 3
00
0 03
2
source
Maxflow algorithm
and reparameterisation
value(flow)=2
0
00
0
)1,1,0(=x
minimum of the energy:
2
0
sink
1 0
0
3
4
3
3
source
Maxflow algorithm
and reparameterisation
Posiform maximisation
Binary variables,
non-submodular functions
Arbitrary functions of binary variables
• Can be solved via maxflow[Hammer,Hansen,Simeone’84][Boros,Hammer,Sun’91]
– Specially constructed graph
• Gives solution to LP relaxation: for each node
xp∈{0, 1/2, 1}
E
LP relaxation
non-negativemaximize
∑∑ ++=qp
qppq
p
ppconst xxxE,
),()()|( θθθθx
Arbitrary functions of binary variables
0
1
0
1
1 1/2 1/2
1/2
1/2
Part of optimal solution
[Hammer, Hansen, Simeone’84]
Graph construction - Main idea
∑
∑
∑
+
+
=
),(~
),(
)(})({
qppq
qppq
ppp
xxE
xxE
xExEunary
pairwise submodular
pairwise non-submodular
• Double # of variables:
– Ideally,
• Write E as a function of both old and new variables
– New function is submodular!
ppp xxx ,→
pp xx −=1
∑
∑
∑
−+−+
−−++
−+=
2
),1(~
)1,(~
2
)1,1(),(
2
)1()(}){},({
qppqppq
qpppppq
pppp
pp
xxExxE
xxExxE
xExExxE
Graph construction - Main idea
∑
∑
∑
+
+
=
),(~
),(
)(})({
qppq
qppq
ppp
xxE
xxE
xExE
• Double # of variables:
– Ideally,
• Write E as a function of both old and new variables
– New function is submodular!
ppp xxx ,→
pp xx −=1
∑
∑
∑
−+−+
−−++
−+=
2
),1(~
)1,(~
2
)1,1(),(
2
)1()(}){},({
qppqppq
qpppppq
pppp
pp
xxExxE
xxExxE
xExExxE
Graph construction - Main idea
∑
∑
∑
+
+
=
),(~
),(
)(})({
qppq
qppq
ppp
xxE
xxE
xExE
• Double # of variables:
– Ideally,
• Write E as a function of both old and new variables
– New function is submodular!
ppp xxx ,→
pp xx −=1
∑
∑
∑
−+−+
−−++
−+=
2
),1(~
)1,(~
2
)1,1(),(
2
)1()(}){},({
qppqppq
qpppppq
pppp
pp
xxExxE
xxExxE
xExExxE
Graph construction - Main idea
∑
∑
∑
+
+
=
),(~
),(
)(})({
qppq
qppq
ppp
xxE
xxE
xExE
• Double # of variables:
– Ideally,
• Write E as a function of both old and new variables
– New function is submodular!
ppp xxx ,→
pp xx −=1
∑
∑
∑
−+−+
−−++
−+=
2
),1(~
)1,(~
2
)1,1(),(
2
)1()(}){},({
qppqppq
qpppppq
pppp
pp
xxExxE
xxExxE
xExExxE
Graph construction - Main idea
∑
∑
∑
+
+
=
),(~
),(
)(})({
qppq
qppq
ppp
xxE
xxE
xExE
• Double # of variables:
– Ideally,
• Write E as a function of both old and new variables
– New function is submodular!
ppp xxx ,→
pp xx −=1
∑
∑
∑
−+−+
−−++
−+=
2
),1(~
)1,(~
2
)1,1(),(
2
)1()(}){},({
qppqppq
qpppppq
pppp
pp
xxExxE
xxExxE
xExExxE
Graph construction - Main idea
∑
∑
∑
+
+
=
),(~
),(
)(})({
qppq
qppq
ppp
xxE
xxE
xExE
• Minimise new function
– Without constraint
}){},({ pp xxE
pp xx −=1
Graph construction
2 00
0 10
sink
source
px
px
Graph construction
2 00
0 10
sink
source
Graph construction
00
0 10
sink
source
2
Graph construction
2 00
0 10
sink
source
px
px
Assigning labels
0
1
=
=
p
p
x
x
• Assign labels based on minimum cut in auxiliary graph
node p is unlabeled
• To maximize # of labeled nodes, choose a particular minimum cut
1
0
=
=
p
p
x
x
Optimality
0 0
0
0
0
0=px
Theorem [Hammer,Hansen,Simeone’84].
Labeling x is part of optimal labeling x*.
1=px ?=px
labeled part unlabeled part
Part B: Lower bound via
convex combination of trees
(⇒⇒⇒⇒ tree-reweighted message passing)
• Goal: compute minimum of the energy for θ :
• In general, intractable!
• Obtaining lower bound:
– Split θ into several components: θ = θ1 + θ2 + ...
– Compute minimum for each component:
– Combine Φ(θ1), Φ(θ2), ... to get a bound on Φ(θ)
• Use trees!
)|(min)( θθ xx
E=Φ
)|(min)( iiE θθ x
x=Φ
Convex combination of trees
[Wainwright, Jaakkola, Willsky ’02]
θ ≡ '
2
1 TθTθ2
1+
graph tree T tree T’
)(θΦ )(2
1 TθΦ )(2
1 'TθΦ+≥
lower bound on the energymaximize
Convex combination of trees
(cont’d)
TRW algorithms
• Goal: find reparameterisation maximizing lower bound
• Apply sequence of different reparameterisation operations:– Node averaging
– Ordinary BP on trees
• Order of operations?– Affects performance dramatically
• Algorithms:– [Wainwright et al. ’02]: parallel schedule
• May not converge
– [Kolmogorov’05]: specific sequential schedule• Lower bound does not decrease, convergence guarantees
• Needs half the memory
Experimental results: stereo
left image ground truth
BP TRW-S20 40 60 80 100
3.6
3.8
4x 10
5
• Global minima for some instances with TRW [Meltzer,Yanover,Weiss’05]
Parts A and B: Summary
• MAP estimation algorithms are based on LP relaxation– Maximize lower bound
• Two ways to formulate lower bound
• Via posiforms: leads to maxflow algorithm (for binary variables)– Polynomial time solution
– Submodular functions: global minimum
– Non-submodular functions: part of optimal solution
• Via convex combination of trees: leads to TRW algorithm– Convergence in the limit (for TRW-S)
– Applicable to arbitrary energy function
Non-binary variables:
Other methods for solving LP
• No polynomial-time algorithm (except general purpose LP solvers)
• Iterative methods:– [Koval,Schlesinger’76]: augmenting DAG algorithm
– [Kovalevsky,Koval’75, Flach’98] (unpublished): max-sum diffusion• See tech. report [Werner’05]
• Not guaranteed to solve LP (only arc constistent solution) – same as TRW
• Special case: submodular functions– LP has integer optimal solution [Schlesinger,Flach’00]
– Reduction to maxflow [Ishikawa’03, D.Schlesinger’05]
EE E
Continuous mincut/maxflow
Continuous mincut/maxflow
• Primal problem:
min ))(( →∫C
dssCg
min|| →∇∫ gu
Alternatively:
subject to
∈=
∈=
tpu
spu
p
p
,1
,0
total variation
[Rudin,Osher,Fatemi’92] :
image restoration
[Amar,Belletini’94]:
definition for arbitrary metric
subject to
Ct
Cs
outside
inside
up = 0
up = 1
Continuous mincut/maxflow
• Primal problem:
min ))(( →∫C
dssCg
min|| →∇∫ gu
Alternatively:
subject to
∈=
∈=
tpu
spu
p
p
,1
,0
subject to
Ct
Cs
outside
inside
• up∈[0,1] – fractional segmentations
• Convex problem
• Integer optimal solution
up = 0
up = 1
Continuous mincut/maxflow
• Dual problem:
(capacity constraint)
(flow conservation)
gf p ≤||r
0div =pfr
subject to
for p ∉ s,t
max )(div →∫s
p dafr
Reparameterisation
• Any flow with
defines reparameterisation
(by the divergence theorem):
tspf p ,for 0div ∉=r
∫∫ ⋅−+=≡CC
dsNfgconstdsgCE )( )(rr
∫=s
dsfconst )(divr
where
Reparameterisation
∫∫ ⋅−+=≡CC
dsNfgconstdsgCE )( )(rr
distance
maps
⇒ non-negativegf ≤||r
lower bound on E(C)
Reparameterisation
∫∫ ⋅−+=≡CC
dsNfgconstdsgCE )( )(rr
Suppose flow saturates cut C*
( for p ∈ C* ):
⇒ C* = minimum cut
Ngf pp
rr=
zero for C*
Global vs. local optimisation algorithms:
Summary
• Geodesic active contours
– Variational approach (e.g. level sets)
• Gradient descent in the space of contours
• Local minimum
• Non-convex formulation
– Graph cuts (e.g. geo-cuts)
• Extended space (fractional segmentations)
• Convex formulation
• Integer solution (for submodular functions)
– `
Global vs. local optimisation algorithms:
Summary
• Geodesic active contours
– Variational approach (e.g. level sets)
• Gradient descent in the space of contours
• Local minimum
• Non-convex formulation
– Graph cuts (e.g. geo-cuts)
• Extended space (fractional segmentations)
• Convex formulation
• Integer solution (for submodular functions)
– `
Other relaxations/extensions
• Energy E(x) defined for integer configurations (xp∈ {0,1})
• How to define for fractional configurations (xp∈ [0,1])?
Other relaxations/extensions
• LP relaxation [Schlesinger’76,Koster et al.’98,Chekuri et al.’00, Wainwright et al.’03]
– Defined for multi-valued variables
– Convex
– E is submodular ⇒ integer solution
• Lovász extension [Lovász’83]
– Defined for binary variables
– Always integer solution
– E is submodular ⇔ extension is convex
– “Submodularity” – discrete analogue of convexity
• Sherali-Adams relaxation, semi-definite relaxation, SOCP relaxation, ...
(0,0)(1,0)
(1,1)
LP relaxation and Lovász extension
Submodular function: E(0,0) + E(1,1) ≤ E(0,1) + E(1,0)
LP relaxation and Lovász extension
(0,0)(1,0)
(1,1)(1,0)
(0,0)(1,0)
(1,1)(1,0)
LP Lovász
Non-submodular function: E(0,0) + E(1,1) ≥ E(0,1) + E(1,0)