MAP Estimation Algorithms in M. Pawan Kumar, University of Oxford Pushmeet Kohli, Microsoft Research...

MAP Estimation Algorithms in

M. Pawan Kumar, University of Oxford

Pushmeet Kohli, Microsoft Research

Computer Vision - Part II

Example: Image Segmentation

E(x) = ∑ ci xi + ∑ cij xi(1-xj)

E: {0,1}n → R0 → fg 1 → bg

Image (D)

i i,jn = number of pixels



E: {0,1}n → R0 → fg 1 → bg

i i,j

Unary Cost (ci)

Dark (negative) Bright (positive)

n = number of pixels



E: {0,1}n → R0 → fg 1 → bg

i i,j

Discontinuity Cost (cij)




E: {0,1}n → R0 → fg 1 → bg

i i,j

Global Minimum (x*)

x* = arg min

E(x) x

How to minimize E(x)?


Outline of the Tutorial

The st-mincut problem

What problems can we solve using st-mincut?

st-mincut based Move algorithms

Recent Advances and Open Problems

Connection between st-mincut and energy

minimization?






minimization?


The st-Mincut Problem

Source

Sink

v1 v2

2

5

9

42

1

Graph (V, E, C)Vertices V = {v1, v2 ... vn}

Edges E = {(v1, v2) ....}

Costs C = {c(1, 2) ....}


Source

Sink

v1 v2

2

5

9

42

1

What is a st-cut?


Source

Sink

v1 v2

2

5

9

42

1

What is a st-cut?

An st-cut (S,T) divides the nodes between source and sink.

What is the cost of a st-cut?

Sum of cost of all edges going from S to T

5 + 2 + 9 = 16


What is a st-cut?

An st-cut (S,T) divides the nodes between source and sink.

What is the cost of a st-cut?

Sum of cost of all edges going from S to T

What is the st-mincut?

st-cut with the minimum cost

Source

Sink

v1 v2

2

5

9

42

1

2 + 1 + 4 = 7

How to compute the st-mincut?

Source

Sink

v1 v2

2

5

9

42

1

Solve the dual maximum flow problem

In every network, the maximum flow equals the cost of the st-

mincut

Min-cut\Max-flow Theorem

Compute the maximum flow between Source and Sink

Constraints

Edges: Flow < Capacity

Nodes: Flow in = Flow out

Maxflow Algorithms

Augmenting Path Based Algorithms

1. Find path from source to sink with positive capacity

2. Push maximum possible flow through this path

3. Repeat until no path can be found

Source

Sink

v1 v2

2

5

9

42

1

Algorithms assume non-negative capacity

Flow = 0

Maxflow Algorithms





Source

Sink

v1 v2

2-2

5-2

9

42

1


Flow = 0 + 2

Maxflow Algorithms

Source

Sink

v1 v2

0

3

9

42

1






Flow = 2

Maxflow Algorithms

Source

Sink

v1 v2

0

3

5

02

1






Flow = 2 + 4

Maxflow Algorithms

Source

Sink

v1 v2

0

3

5

02

1






Flow = 6

Maxflow Algorithms

Source

Sink

v1 v2

0

2

4

02+1

1-1






Flow = 6 + 1

Maxflow Algorithms

Source

Sink

v1 v2

0

2

4

03

0






Flow = 7

History of Maxflow Algorithms

[Slide credit: Andrew Goldberg]

Augmenting Path and Push-Relabel

n: #nodes

m: #edges

U: maximum edge weight

Algorithms assume non-negative edge

weights

Augmenting Path based Algorithms

a1 a2

1000 1

Sink

Source

1000

1000

1000

0

Ford Fulkerson: Choose any augmenting path

a1 a2

1000 1

Sink

Source

1000

1000

1000

0


Bad Augmenting

Paths


a1 a2

1000 1

Sink

Source

1000

1000

1000

0


Bad Augmenting

Path


a1 a2

9990

Sink

Source

1000

1000

9991



a1 a2

9990

Sink

Source

1000

1000

9991


n: #nodes

m: #edges

We will have to perform 2000 augmentations!

Worst case complexity: O (m x Total_Flow)

(Pseudo-polynomial bound: depends on flow)


Dinic: Choose shortest augmenting path

n: #nodes

m: #edges

Worst case Complexity: O (m n2)


a1 a2

1000 1

Sink

Source

1000

1000

1000

0

Maxflow in Computer Vision

• Specialized algorithms for vision problems– Grid graphs – Low connectivity (m ~ O(n))

• Dual search tree augmenting path algorithm[Boykov and Kolmogorov PAMI 2004]• Finds approximate shortest

augmenting paths efficiently• High worst-case time complexity• Empirically outperforms other

algorithms on vision problems

Maxflow in Computer Vision

• Specialized algorithms for vision problems– Grid graphs – Low connectivity (m ~ O(n))

• Dual search tree augmenting path algorithm[Boykov and Kolmogorov PAMI 2004]• Finds approximate shortest

augmenting paths efficiently• High worst-case time complexity• Empirically outperforms other

algorithms on vision problems• Efficient code available on the

webhttp://www.adastral.ucl.ac.uk/~vladkolm/software.html






minimization?


St-mincut and Energy Minimization

T

S st-mincut

E: {0,1}n → R

Minimizing a Qudratic

Pseudoboolean function E(x)

Functions of boolean variables

Pseudoboolean?

Polynomial time st-mincut algorithms require non-negative

edge weights

E(x) = ∑ ci xi + ∑ cij xi(1-xj) cij≥0i,ji

So how does this work?

Construct a graph such that:

1.Any st-cut corresponds to an assignment of x

2.The cost of the cut is equal to the energy of x : E(x)

SolutionT

S st-mincut

E(x)

Graph Construction

Sink (1)

Source (0)

a1 a2

E(a1,a2)

Graph Construction

Sink (1)

Source (0)

a1 a2

E(a1,a2) = 2a1

2

Graph Construction

a1 a2

E(a1,a2) = 2a1 + 5ā1

2

5

Sink (1)

Source (0)

Graph Construction

a1 a2

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2

2

5

9

4

Sink (1)

Source (0)

Graph Construction

a1 a2

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2

2

5

9

4

2

Sink (1)

Source (0)

Graph Construction

a1 a2

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

2

5

9

4

2

1

Sink (1)

Source (0)

Graph Construction

a1 a2

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

2

5

9

4

2

1a1 = 1 a2 = 1

E (1,1) = 11

Cost of cut = 11

Sink (1)

Source (0)

Graph Construction

a1 a2

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

2

5

9

4

2

1

Sink (1)

Source (0)

a1 = 1 a2 = 0

E (1,0) = 8

st-mincut cost = 8

Energy Function Reparameterization

Two functions E1 and E2 are reparameterizations if

E1 (x) = E2 (x) for all x

For instance:

E1 (a1) = 1+ 2a1 + 3ā1

E2 (a1) = 3 + ā1

a1 ā1 1+ 2a1 + 3ā1

3 + ā1

0 1 4 4

1 0 3 3

Flow and Reparametrization

a1 a2

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

2

5

9

4

2

1

Sink (1)

Source (0)


a1 a2

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

2

5

9

4

2

1

Sink (1)

Source (0)

2a1 + 5ā1

= 2(a1+ā1) + 3ā1

= 2 + 3ā1


Sink (1)

Source (0)

a1 a2

E(a1,a2) = 2 + 3ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

0

3

9

4

2

12a1 + 5ā1

= 2(a1+ā1) + 3ā1

= 2 + 3ā1

Sink (0)

Source (1)

a1 a2

E(a1,a2) = 2 + 3ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

0

3

9

4

2

1


9a2 + 4ā2

= 4(a2+ā2) + 5ā2

= 4 + 5ā2

Sink (0)

Source (1)

a1 a2

E(a1,a2) = 2 + 3ā1+ 5a2 + 4 + 2a1ā2 + ā1a2

0

3

5

0

2

19a2 + 4ā2

= 4(a2+ā2) + 5ā2

= 4 + 5ā2


Sink (0)

Source (1)

a1 a2

E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2

0

3

5

0

2

1


Sink (0)

Source (1)

a1 a2

E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2

0

3

5

0

2

1


3ā1+ 5a2 + 2a1ā2

= 2(ā1+a2+a1ā2) +ā1+3a2= 2(1+ā1a2) +ā1+3a2

a1 a2 F1 F2

0 0 1 1

0 1 2 2

1 0 1 1

1 1 1 1

F1 = ā1+a2+a1ā2

F2 = 1+ā1a2

Sink (0)

Source (1)

a1 a2

E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2

0

1

3

0

0

3


3ā1+ 5a2 + 2a1ā2

= 2(ā1+a2+a1ā2) +ā1+3a2= 2(1+ā1a2) +ā1+3a2

a1 a2 F1 F2

0 0 1 1

0 1 2 2

1 0 1 1

1 1 1 1

F1 = ā1+a2+a1ā2

F2 = 1+ā1a2

Sink (0)

Source (1)

a1 a2

0

1

3

0

0

3


E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2

No more augmenting paths

possible

Sink (0)

Source (1)

a1 a2

0

1

3

0

0

3


E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2

Total Flow

Residual Graph(positive

coefficients)

bound on the optimal solution

Inference of the optimal solution becomes trivial because the bound is

tight

Sink (0)

Source (1)

a1 a2

0

1

3

0

0

3


E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2

a1 = 1 a2 = 0

E (1,0) = 8

st-mincut cost = 8Total Flow

bound on the optimal solution

Inference of the optimal solution becomes trivial because the bound is

tight

Residual Graph(positive

coefficients)



E: {0,1}n → R0 → fg 1 → bg

i i,j

Global Minimum (x*)

x* = arg min

E(x) x

How to minimize E(x)?

How does the code look like?

Sink (1)

Source (0)

Graph *g;

For all pixels p

/* Add a node to the graph */nodeID(p) = g->add_node();

/* Set cost of terminal edges */set_weights(nodeID(p), fgCost(p), bgCost(p));

end

for all adjacent pixels p,qadd_weights(nodeID(p), nodeID(q), cost);

end

g->compute_maxflow();

label_p = g->is_connected_to_source(nodeID(p));

// is the label of pixel p (0 or 1)


Graph *g;

For all pixels p



end

for all adjacent pixels p,qadd_weights(nodeID(p), nodeID(q), cost);

end




a1 a2

fgCost(a1)

Sink (1)

Source (0)

fgCost(a2)

bgCost(a1) bgCost(a2)

Graph *g;

For all pixels p



end

for all adjacent pixels p,qadd_weights(nodeID(p), nodeID(q), cost(p,q));

end





a1 a2

fgCost(a1)

Sink (1)

Source (0)

fgCost(a2)


cost(p,q)

Graph *g;

For all pixels p



end

for all adjacent pixels p,qadd_weights(nodeID(p), nodeID(q), cost(p,q));

end





a1 a2

fgCost(a1)

Sink (1)

Source (0)

fgCost(a2)


cost(p,q)

a1 = bg a2 = fg

Image Segmentation in Video

Image Flow Global Optimum

pqw

n-links st-cuts

t

= 0

= 1

E(x) x*

Image Segmentation in Video

Image

FlowGlobal Optimu

m

Dynamic Energy Minimization

EB SBcomputationally

expensive operation

EA SA

minimize

Recycling

Solutions

Can we do better?

Boykov & Jolly ICCV’01, Kohli & Torr (ICCV05, PAMI07)


EB SBcomputationally

expensive operation

EA SA

minimize

cheaperoperation

Simplerenergy

EB*

differencesbetweenA and B

A and Bsimilar

Kohli & Torr (ICCV05, PAMI07)

3 – 100000 time

speedup!

Reparametrization

Reuse flow


Reparametrized Energy


Kohli & Torr (ICCV05, PAMI07)


E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2

Original Energy

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 7a1ā2 + ā1a2

E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2 + 5a1ā2

New Energy

New Reparametrized Energy






minimization?


Minimizing Energy Functions

Space of Function Minimization

Problems

Submodular

Functions

NP-Hard

• General Energy Functions– NP-hard to minimize– Only approximate

minimization possible

• Easy energy functions – Solvable in polynomial time– Submodular ~ O(n6)

MAXCUT

Functions defined on trees

Submodular Set Functions

Set function f 2|E| ℝ

2|E| = #subsets of E

Let E= {a1,a2, .... an} be a set

Submodular Set Functions

Set function f 2|E| ℝ is submodular if

E

A B

f(A) + f(B) f(AB) + f(AB) for all A,B E

2|E| = #subsets of E

Let E= {a1,a2, .... an} be a set

Important Property

Sum of two submodular functions is submodular

Minimizing Submodular Functions

• Minimizing general submodular functions– O(n5 Q + n6) where Q is function evaluation time

[Orlin, IPCO 2007]

• Symmetric submodular functions– E (x) = E (1 - x)– O(n3) [Queyranne 1998]

• Quadratic pseudoboolean– Can be transformed to st-mincut– One node per variable [ O(n3) complexity]– Very low empirical running time

Submodular Pseudoboolean Functions

• All functions for one boolean variable (f: {0,1} -> R) are submodular

• A function of two boolean variables (f: {0,1}2 -> R) is submodular if

f(0,1) + f(1,0) f(0,0) + f(1,1)

• A general pseudoboolean function f 2n ℝ is submodular if all its projections fp are submodular i.e.fp(0,1) + fp (1,0) fp (0,0) + fp

(1,1)

Function defined over boolean vectors x = {x1,x2, .... xn}

Definition:

E(x) = ∑ θi (xi) + ∑ θij (xi,xj)i,ji

Quadratic Submodular Pseudoboolean Functions

θij(0,1) + θij (1,0) θij

(0,0) + θij

(1,1)For all ij



θij(0,1) + θij (1,0) θij

(0,0) + θij

(1,1)For all ij

E(x) = ∑ ci xi + ∑ cij xi(1-xj) cij≥0i,ji

Equivalent (transformable)

i.e. All submodular QPBFs are st-mincut solvable

A B

C D

0 1

0

1

xi

xj

= A +0 0

C-A C-A

0 1

0

1

0 D-C

0 D-C

0 1

0

1

0B+C-A-D

0 0

0 1

0

1

+ +

if x1=1 add

C-A

if x2 = 1 add

D-C

B+C-A-D 0 is true from the submodularity of θij

How are they equivalent?

A = θij (0,0) B = θij(0,1) C = θij

(1,0) D = θij

(1,1)

θij (xi,xj) = θij(0,0)

+ (θij(1,0)-θij(0,0)) xi + (θij(1,0)-θij(0,0)) xj

+ (θij(1,0) + θij(0,1) - θij(0,0) - θij(1,1)) (1-xi) xj



θij(0,1) + θij (1,0) θij

(0,0) + θij

(1,1)For all ij

Equivalent (transformable)

T

Sst-mincut

x in {0,1}n

Minimizing Non-Submodular Functions

• Minimizing general non-submodular functions is NP-hard.

• Commonly used method is to solve a relaxation of the problem


θij(0,1) + θij (1,0) ≤ θij

(0,0) + θij (1,1) for some ij

[Slide credit: Carsten Rother]

pairwise nonsubmodular

unary

pairwise submodular

Minimization using Roof-dual Relaxation

)0,1()1,0()1,1()0,0( pqpqpqpq

)0,1(~

)1,0(~

)1,1(~

)0,0(~

pqpqpqpq

),(~

),(

)(})({

qppq

qppq

ppp

xx

xx

xxE


2

),1(~

)1,(~

2

)1,1(),(

2

)1()(}){},({'

qppqqppq

qppqqppq

pppppp

xxxx

xxxx

xxxxE

Double number of variables:

),(~

),(

)(})({

qppq

qppq

ppp

xx

xx

xxE

ppp xxx ,


)1( pp xx


2

),1(~

)1,(~

2

)1,1(),(

2

)1()(}){},({'

qppqqppq

qppqqppq

pppppp

xxxx

xxxx

xxxxE


),(~

),(

)(})({

qppq

qppq

ppp

xx

xx

xxE

ppp xxx ,


)1,(~

),( qppqqppq xxxxf

)0,1(~

)1,0(~

)1,1(~

)0,0(~

pqpqpqpq )1,1()0,0()0,1()1,0( pqpqpqpq ffff

Non- submodular Submodular

)1( pp xx

2

),1(~

)1,(~

2

)1,1(),(

2

)1()(}){},({'

qppqqppq

qppqqppq

pppppp

xxxx

xxxx

xxxxE


),(~

),(

)(})({

qppq

qppq

ppp

xx

xx

xxE

ppp xxx ,


)1( pp xx

•

• is submodular ! Ignore (solvable using st-mincut)

Property of the problem:

pp xx 1

pp xx 1 is the optimal labelpx

Property of the solution:

2

),1(~

)1,(~

2

)1,1(),(

2

)1()(}){},({'

qppqqppq

qppqqppq

pppppp

xxxx

xxxx

xxxxE


),(~

),(

)(})({

qppq

qppq

ppp

xx

xx

xxE

ppp xxx , )1( pp xx


Recap

• Exact minimization of Submodular QBFs using graph cuts.

• Obtaining partially optimal solutions of non-submodular QBFs using graph cuts.

But ...

• Need higher order energy functions to model image structure – Field of experts [Roth and Black]

• Many problems in computer vision involve multiple labels

E(x) = ∑ θi (xi) + ∑ θij (xi,xj) + ∑ θc (xc)i,ji c

x ϵ Labels L = {l1, l2, … , lk} Clique c V

Transforming problems in QBFs

Multi-label Functions

Pseudoboolean Functions

Higher order Pseudoboolean

Functions

Quadratic Pseudoboolean

Functions

Higher order to Quadratic

• Simple Example using Auxiliary variables

{0 if all xi = 0C1 otherwise

f(x) =

min f(x) min C1a + C1 ∑ ā xi

x ϵ L = {0,1}n

x =x,a ϵ {0,1}

Higher Order Submodular

Function

Quadratic Submodular Function

∑xi = 0 a=0 (ā=1) f(x) = 0

∑xi ≥ 1 a=1 (ā=0) f(x) = C1


min f(x) min C1a + C1 ∑ ā xix=

x,a ϵ {0,1}


Function


∑xi

1 2 3

C1

C1∑xi


min f(x) min C1a + C1 ∑ ā xix=

x,a ϵ {0,1}


Function


∑xi

1 2 3

C1

C1∑xi

a=1a=0Lower

envelop of concave

functions is concave


min f(x) min f1 (x)a + f2(x)āx

=x,a ϵ {0,1}


Function


∑xi

1 2 3

a=1

Lower envelop of concave


f2(x)

f1(x)


min f(x) min f1 (x)a + f2(x)āx

=x,a ϵ {0,1}


Function


∑xi

1 2 3

a=1a=0Lower

envelop of concave


f2(x)

f1(x)

Transforming problems in QBFs

Multi-label Functions

Pseudoboolean Functions

Higher order Pseudoboolean

Functions

Quadratic Pseudoboolean

Functions

Multi-label to Pseudo-boolean

So what is the problem?

Eb (x1,x2, ..., xm)Em (y1,y2, ..., yn)

Multi-label Problem

Binary label Problem

yi ϵ L = {l1, l2, … , lk} xi ϵ L = {0,1}

such that:Let Y and X be the set of feasible solutions, then

1.For each binary solution x ϵ X with finite energy there exists exactly one multi-label solution y ϵ Y

-> One-One encoding function T:X->Y

2. arg min Em(y) = T(arg min Eb(x))


• Popular encoding scheme [Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]



Ishikawa’s result:

E(y) = ∑ θi (yi) + ∑ θij (yi,yj)i,ji

y ϵ Labels L = {l1, l2, … , lk}

θij (yi,yj) = g(|yi-yj|)Convex Functio

n

g(|yi-yj|)

|yi-yj|



Schlesinger & Flach ’06:E(y) = ∑ θi (yi) + ∑ θij (yi,yj)

i,ji

y ϵ Labels L = {l1, l2, … , lk}

θij(li+1,lj) + θij (li,lj+1) θij

(li,lj) + θij

(li+1,lj+1)

Covers all Submodular multi-label functions

More general than Ishikawa


• Applicability– Only solves restricted class of energy functions– Cannot handle Potts model potentials

• Computational Cost– Very high computational cost– Problem size = |Variables| x |Labels|– Gray level image denoising (1 Mpixel image)

(~2.5 x 108 graph nodes)

Problems






minimization?


St-mincut based Move algorithms

• Commonly used for solving non-submodular multi-label problems

• Extremely efficient and produce good solutions

• Not Exact: Produce local optima


x ϵ Labels L = {l1, l2, … , lk}

Move Making Algorithms

Solution Space

En

erg

y

Move Making Algorithms

Search Neighbourhood

Current Solution

Optimal Move

Solution Space

En

erg

y

Computing the Optimal Move


Current Solution

Optimal Move

xc

(t) Key Property

Move Space

Bigger move space

Solution Space

En

erg

y

• Better solutions

• Finding the optimal move hard

Moves using Graph Cuts

Expansion and Swap move algorithms

[Boykov Veksler and Zabih, PAMI 2001]

• Makes a series of changes to the solution (moves)

• Each move results in a solution with smaller energy

Space of Solutions (x) : LN

Move Space (t) : 2N


Current Solution

N Number of Variables

L Number of Labels

Moves using Graph Cuts

Expansion and Swap move algorithms

[Boykov Veksler and Zabih, PAMI 2001]

• Makes a series of changes to the solution (moves)

• Each move results in a solution with smaller energy Current SolutionCurrent Solution

Construct a move function

Construct a move function

Minimize move function to get optimal move

Minimize move function to get optimal move

Move to new

solution

Move to new

solution

How to minimize

move functions?

General Binary Moves

Minimize over move variables t to get the optimal move

x = t x1 + (1- t) x2

New solution

Current Solution

Second solution

Em(t) = E(t x1 + (1- t) x2)

Boykov, Veksler and Zabih, PAMI 2001

Move energy is a submodular QPBF (Exact Minimization

Possible)

Swap Move• Variables labeled α, β can swap their

labels

[Boykov, Veksler, Zabih]

Swap Move

Sky

House

Tree

GroundSwap Sky, House


• Variables labeled α, β can swap their labels

Swap Move• Variables labeled α, β can swap their

labels

• Move energy is submodular if:– Unary Potentials: Arbitrary– Pairwise potentials: Semimetric


θij (la,lb) ≥ 0

θij (la,lb) = 0 a = b

Examples: Potts model, Truncated Convex

Expansion Move

[Boykov, Veksler, Zabih][Boykov, Veksler, Zabih]

• Variables take label or retain current label

Expansion Move

Sky

House

Tree

Ground

Initialize with TreeStatus: Expand GroundExpand HouseExpand Sky



Expansion Move

• Move energy is submodular if:– Unary Potentials: Arbitrary

– Pairwise potentials: Metric


θij (la,lb) + θij (lb,lc) ≥ θij (la,lc)

Semi metric+

Triangle Inequality


Examples: Potts model, Truncated linear

Cannot solve truncated quadratic

General Binary Moves

Move Type First Solution

Second Solution

Guarantee

Expansion Old solution All alpha Metric

Fusion Any solution Any solution

Minimize over move variables t

x = t x1 + (1-t) x2

New solution

First solution

Second solution

Move functions can be non-submodular!!

Solving Continuous Problems using Fusion Move

x = t x1 + (1-t) x2

(Lempitsky et al. CVPR08, Woodford et al. CVPR08)

x1, x2 can be continuous

Fx1

x2

x

Optical Flow Example

Final Solutio

n

Solution from

Method 1

Solution from

Method 2

Range Moves • Move variables can be multi-label

• Optimal move found out by using the Ishikawa

• Useful for minimizing energies with truncated convex pairwise potentials

O. Veksler, CVPR 2007

θij (yi,yj) = min(|yi-yj|,T)

|yi-yj|θij (yi,yj)

T

x = (t ==1) x1 + (t==2) x2 +… +(t==k) xk

Move Algorithms for Solving Higher Order Energies

• Higher order functions give rise to higher order move energies

• Move energies for certain classes of higher order energies can be transformed to QPBFs.

E(x) = ∑ θi (xi) + ∑ θij (xi,xj) + ∑ θc (xc)i,ji c

x ϵ Labels L = {l1, l2, … , lk} Clique c V

[Kohli, Kumar and Torr, CVPR07] [Kohli, Ladicky and Torr, CVPR08]






minimization?


Solving Mixed Programming Problems

x – binary image segmentation (xi ∊ {0,1})

ω – non-local parameter (lives in some large set Ω)

constant unary potentia

ls

pairwise

potentials

E(x,ω) = C(ω) + ∑ θi (ω, xi) + ∑ θij (ω,xi,xj)i,ji

≥ 0

Rough Shape Prior

Stickman Model

ωPose

θi (ω, xi) Shape Prior

Open Problems

Submodular

Functions

st-mincut

Equivalent

Characterization of Problems Solvable using st-mincut

• What functions can be transformed to submodular QBFs?

Minimizing General Higher Order Functions

• We saw how simple higher order potentials can be solved

• How more sophisticated higher order potentials can be solved?

Exact Transformation

(global optimum)

Or Relaxed transformation

(partially optimal)

Summary

T

Sst-mincut

Labelling

Problem

Submodular Quadratic Pseudoboolean

Function

Move making algorithms

Sub-problem

Thanks. Questions?

Use of Higher order Potentials

Stereo - Woodford et al. CVPR 2008

P1 P2 P3

5

6

7

8

Pixels

Disparity Labels

E(x1,x2,x3) = θ12 (x1,x2) + θ23 (x2,x3)

θij (xi,xj) =0 if xi=xj

C otherwise{

E(6,6,6) = 0 + 0 = 0



P1 P2 P3

5

6

7

8

Pixels

Disparity Labels

E(x1,x2,x3) = θ12 (x1,x2) + θ23 (x2,x3)


C otherwise{

E(6,6,6) = 0 + 0 = 0

E(6,7,7) = 1 + 0 = 1



P1 P2 P3

5

6

7

8

Pixels

Disparity Labels

Pairwise potential penalize slanted planar surfaces

E(x1,x2,x3) = θ12 (x1,x2) + θ23 (x2,x3)


C otherwise{

E(6,6,6) = 0 + 0 = 0

E(6,7,7) = 1 + 0 = 1

E(6,7,8) = 1 + 1 = 2



Current Solution

Optimal Move

x

E(x)

xcTransformation function

T(t)

T(xc, t) = xn = xc + t



Current Solution

Optimal Move

E(x)

xcTransformation function

T

Em Move Energy

(t)

x

Em(t) = E(T(xc, t))

T(xc, t) = xn = xc + t



Current Solution

Optimal Move

E(x)

xc

Em(t) = E(T(xc, t))

Transformation function

T

Em Move Energy

T(xc, t) = xn = xc + t minimize

t*Optimal Move

(t)

x

MAP Estimation Algorithms in M. Pawan Kumar, University of Oxford Pushmeet Kohli, Microsoft Research...

Documents

Transcript of MAP Estimation Algorithms in M. Pawan Kumar, University of Oxford Pushmeet Kohli, Microsoft Research...