Lecture 4

C25 Optimization Hilary 2013 A. Zisserman

Optimization on graphs

• Max-flow & min-cut

• The augmented path algorithm

• Optimization for binary image graphs

• Applications

Max FlowGiven: a weighted directed graph with two distinguished nodes:

• source s, • sink (destination) t

Interpret edge weights (all positive) as capacities

Goal: Find maximum flow from s to t

• Flow does not exceed capacity in any edge

• Flow at every vertex satisfies equilibrium[ flow in equals flow out ]

e.g. oil flowing through pipes, internet routing

B1 reminder

Example 3 cont.

Slide: Robert Sedgewick and Kevin Wayne

B1 reminder

Example 2 cont.B1 reminder

TS

1

2

1

2 1

3

2

Max flow example

• max flow is unique = 2

• but there may be multiple paths (solutions) that achieve it

Matlab LP function linprog for max-flow

>> f = [ -1; -1; 0; 0; 0 ];

>> A = [1 0 0 0 00 1 0 0 00 0 1 0 00 0 0 1 00 0 0 0 1 ];

>> b = [1; 2; 2; 3; 1];

>> Aeq = [1 0 -1 -1 0 0 1 1 0 -1];

>> beq = [ 0; 0 ];

>> lb = zeros(5,1);

>> x = linprog( f, A, b, Aeq, beq, lb );

TS

1

2

1

2 1

3

2

>> Optimization terminated.

>> x

x = 1.0 1.0 0.0 1.0 1.0

>> flow = - x’ * f

flow = 2.0

e1

e2

e3e4

e5

e1 e2 e3 e4 e5

Another example

Source

Sink

v1 v2

2/2

3/5

5/9

4/43

1/1

Max Flow = 7

Source

Sink

v1 v2

2

5

9

42

1

The st-Mincut Problem

Source

Sink

v1 v2

2

5

9

42

1

Slides from Pushmeet Kohli

• Each node is either assigned to the source S or sink T

• The cost of the edge (i, j) is taken if (i∈S) and (j∈T)

•An st-cut (S,T) divides the nodes between source and sink

• The cost of the cut is the sum of costs of all edges going from S to T

• The st-min-cut is the cut with lowest cost

The st-Mincut Problem

Source

Sink

v1 v2

2

5

9

42

1

5 + 2 + 9 = 16

• Each node is either assigned to the source S or sink T

• The cost of the edge (i, j) is taken if (i∈S) and (j∈T)

•An st-cut (S,T) divides the nodes between source and sink

• The cost of the cut is the sum of costs of all edges going from S to T

• The st-min-cut is the cut with lowest cost

The st-Mincut Problem

Source

Sink

v1 v2

2

5

9

42

1

2 + 1 + 4 = 7

• Each node is either assigned to the source S or sink T

• The cost of the edge (i, j) is taken if (i∈S) and (j∈T)

•An st-cut (S,T) divides the nodes between source and sink

• The cost of the cut is the sum of costs of all edges going from S to T

• The st-min-cut is the cut with lowest cost

Min-cut\Max-flow Theorem

Source

Sink

v1 v2

2

5

9

42

1

In every network, the maximum flow equals the cost of the st-mincut

Max flow = min cut = 7

Next: the augmented path algorithm for computing the max-flow/min-cut

Maxflow Algorithms

Augmenting Path Based Algorithms

1. Find path from source to sink with positive capacity

2. Push maximum possible flow through this path

3. Repeat until no path can be found

Source

Sink

v1 v2

2

5

9

42

1

Algorithms assume non-negative capacity

Flow = 0

Maxflow Algorithms

Augmenting Path Based Algorithms

1. Find path from source to sink with positive capacity

2. Push maximum possible flow through this path

3. Repeat until no path can be found

Source

Sink

v1 v2

2

5

9

42

1

Algorithms assume non-negative capacity

Flow = 0

Maxflow Algorithms

Augmenting Path Based Algorithms

1. Find path from source to sink with positive capacity

2. Push maximum possible flow through this path

3. Repeat until no path can be found

Source

Sink

v1 v2

2-2

5-2

9

42

1

Algorithms assume non-negative capacity

Flow = 0 + 2

Maxflow Algorithms

Source

Sink

v1 v2

0

3

9

42

1

Augmenting Path Based Algorithms

1. Find path from source to sink with positive capacity

2. Push maximum possible flow through this path

3. Repeat until no path can be found

Algorithms assume non-negative capacity

Flow = 2

Maxflow Algorithms

Source

Sink

v1 v2

0

3

9

42

1

Augmenting Path Based Algorithms

1. Find path from source to sink with positive capacity

2. Push maximum possible flow through this path

3. Repeat until no path can be found

Algorithms assume non-negative capacity

Flow = 2

Maxflow Algorithms

Source

Sink

v1 v2

0

3

9

42

1

Augmenting Path Based Algorithms

1. Find path from source to sink with positive capacity

2. Push maximum possible flow through this path

3. Repeat until no path can be found

Algorithms assume non-negative capacity

Flow = 2

Maxflow Algorithms

Source

Sink

v1 v2

0

3

5

02

1

Augmenting Path Based Algorithms

1. Find path from source to sink with positive capacity

2. Push maximum possible flow through this path

3. Repeat until no path can be found

Algorithms assume non-negative capacity

Flow = 2 + 4

Maxflow Algorithms

Source

Sink

v1 v2

0

3

5

02

1

Augmenting Path Based Algorithms

1. Find path from source to sink with positive capacity

2. Push maximum possible flow through this path

3. Repeat until no path can be found

Algorithms assume non-negative capacity

Flow = 6

Maxflow Algorithms

Source

Sink

v1 v2

0

3

5

02

1

Augmenting Path Based Algorithms

1. Find path from source to sink with positive capacity

2. Push maximum possible flow through this path

3. Repeat until no path can be found

Algorithms assume non-negative capacity

Flow = 6

Maxflow Algorithms

Source

Sink

v1 v2

0

2

4

02+1

1-1

Augmenting Path Based Algorithms

1. Find path from source to sink with positive capacity

2. Push maximum possible flow through this path

3. Repeat until no path can be found

Algorithms assume non-negative capacity

Flow = 6 + 1

Maxflow Algorithms

Source

Sink

v1 v2

0

2

4

03

0

Augmenting Path Based Algorithms

1. Find path from source to sink with positive capacity

2. Push maximum possible flow through this path

3. Repeat until no path can be found

Algorithms assume non-negative capacity

Flow = 7

Maxflow Algorithms

Source

Sink

v1 v2

0

2

4

03

0

Flow = 7

Source

Sink

v1 v2

2

5

9

42

1

2 + 1 + 4 = 7

Min‐cut = 7

Image Graphs

pixels

• many loops

• very large number of nodes (variables) – millions

• dynamic programming can’t be used

Consider binary graphs (h = 2)

ji

Example: noise removal in an image

• n is the number of pixels in the image, e.g. n = 1 M

• Graph structure: each vertex is connected to four neighbours

1

0

i

N (i)

f(x) =Xi

(zi−xi)2+dφ(xi, xi−1)

noisy data

zi = xi+ wi

where

xi ∈ {0,1} wi ∼ N(0,σ2)and

φ(xi, xj) =

(0 ifxi = xj1 ifxi 6= xj.

Graph cut algorithmsBinary optimization: each variable x has one of two possible values

f(x) =nXi=1

{mi(xi) +X

j∈N(i)φi(xi, xj)}

If f(x) is sub-modular, then it can be optimized by the Min-Cut algorithm

xi ∈ {0,1}

The cost function f(x) is sub-modular if

φ(0,0) + φ(1,1) <= φ(0,1) + φ(1,0)

N (i) is the neighbourhood of node i i

Complexity of minimization:

• exhaustive search O(2n)

• min cut O(n3)

original

original plus noise

Min x

20 40 60 80 100 120 140 160 180 200 220

20

40

60

20 40 60 80 100 120 140 160 180 200 220

20

40

60

20 40 60 80 100 120 140 160 180 200 220

20

40

60

d = 10

20 40 60 80 100 120 140 160 180 200 220

20

40

60

20 40 60 80 100 120 140 160 180 200 220

20

40

60

20 40 60 80 100 120 140 160 180 200 220

20

40

60

original

original plus noise

Min x

d = 60

n ∼ 20 K

Optimization using graph cuts

Stage 1: map the cost function f(x) onto a flow network so that a cut of the network corresponds to the cost f(x)

Stage 2: compute the min-cut of the network using an augmented path algorithm

pqw

n-links a cuts

t

Map f(x) onto network flow

Construct a network so that a cut corresponds to an assignment of xi

m1(1) D

C B

A

x1 x2

Label ‘0’

Label ‘1’ m2(1)

m1(0) m2(0)

= 0

= 1

f(x) =nXi=1

{mi(xi) +X

j∈N(i)φi(xi, xj)}

m1(1) D

C B

A

x1 x2

m2(1)

m1(0) m2(0)

Sink (1)

x1

Source (0)

m1(1)

m1(0)

m2(1)

m2(0)

For unary terms only:

Sink (1)

x1 x2

Source (0)

m1(1)

m1(0)

m2(1)

m2(0)

x2

x1 = 1, x2 = 1

→ f(x) = m1(1) +m2(1)

cut

Sink (1)

x1 x2

Source (0)

m1(1) + C - A

m1(0)

m2(1) + D - C

m2(0)

DC

BA

m1(1) D

C B

A

x1 x2

m2(1)

m1(0) m2(0)

0 1

0

1

x1

x2

= A +C-AC-A

00

0 1

0

1 D-C0

D-C0

0 1

0

1 00

B+C-A-D0

0 1

0

1

+ +

B+C-A-D

Now, include pair wise term …

add C-A if x1 = 1

add D-C if x2 = 1

Sub-modular constraint: flows must be positive. So, B+C-A-D >= 0

Sink (1)

x1x2

Source (0)

m1(1) + C - A

m1(0)

m2(1) + D - C

m2(0)

B+C-A-Dx1 = 1, x2 = 1

→ f(x) = m1(1) +m2(1) +D − A

m1(1) D

C B

A

x1 x2

m1(0)

cut

Sink (1)

x1x2

Source (0)

m1(1) + C - A

m1(0)

m2(1) + D - C

m2(0)

B+C-A-Dx1 = 0, x2 = 1

→ f(x) = m1(0) +m2(1)

+D − C +B+ C − A−D= m1(0) +m2(1) +B −A

m1(1) D

C B

A

x1 x2

m1(0)

cut

Summary: optimization using graph cuts

Stage 1: map the cost function f(x) onto a flow network so that a cut of the network corresponds to the cost f(x)

Stage 2: compute the min-cut of the network using an augmented path algorithm

Applications

Optimization of binary image graph using graph-cuts:

1. Image cut-out and editing

2. Image quilting

3. Interactive Digital Photo-montage

1. Image cut-out by binary segmentation

Object - white, Background - green/grey Graph G = (V,E)

Each vertex corresponds to a pixel

Edges define a 4-neighbourhood grid graph

Assign a label to each vertex from L = {obj,bkg}

Graph G = (V,E)

Cost of a labelling f : V L Per Vertex Cost

Cost of label ‘obj’ low Cost of label ‘bkg’ high

Object - white, Background - green/grey

Graph G = (V,E)

Cost of a labelling f : V L

Cost of label ‘obj’ high Cost of label ‘bkg’ low

Per Vertex Cost

UNARY COST

Object - white, Background - green/grey

Graph G = (V,E)

Cost of a labelling f : V L Per Edge Cost

Cost of same label low

Cost of different labels high

Object - white, Background - green/grey

Graph G = (V,E)

Cost of a labelling f : V L

Cost of different labels low

Per Edge Cost

PAIRWISECOST

Object - white, Background - green/grey

Graph G = (V,E)

Problem: Find the labelling with minimum cost f*

Object - white, Background - green/grey

f(x) =nXi=1

{mi(xi) +X

j∈N(i)φi(xi, xj)}jiN (i)

• xi = 1 for foreground pixels, xi = 0 for background

• mi(xi) is likelihood that pixel at i is foreground (if xi = 1), or back-

ground (if xi = 0 ), e.g. using colour histogram of seed regions

• φ(xi, xj) penalizes a change of state:

φi(xi, xj) =

(0 ifxi = xj

γe−β(Ii−Ij)2ifxi 6= xj.

Application: foreground/background image segmentation

foreground Seed Pixels

Background Seed Pixels

use seed pixels to learn colour distribution

Image editing …

Available in Microsoft Office …

Image Quilting

Example: Texture Synthesis

Goal of Texture Synthesis: create new samples of a given texture

Many applications: virtual environments, hole-filling, texturing surfaces

Input texture

B1 B2

Random placement of blocks

block

B1 B2

Neighboring blocksconstrained by overlap

B1 B2

Minimal errorboundary cut

Algorithm

• Pick size of block and size of overlap

• Synthesize blocks in raster order

• Search input texture for block that satisfies overlap constraints (above and left)

• Paste new block into resulting texture> use graph cuts to compute minimal error boundary cut

Efros & Freeman 2001, Kwatra et al. 2003

min. error boundary

Minimal error boundary

overlapping blocks vertical boundary

__ ==22

overlap error

Interactive Digital Photomontage

Agarwala et al. 2004

Use graph-cuts to quilt images