MIT

Randomization in Graph Optimization

ProblemsDavid Karger

MIT

http://theory.lcs.mit.edu/~karger

MIT

Randomized Algorithms

Flip coins to decide what to do next Avoid hard work of making “right” choice Often faster and simpler than

deterministic algorithms

Different from average-case analysis» Input is worst case» Algorithm adds randomness

MIT

Methods

Random selection» if most candidate choices “good”, then a

random choice is probably good Monte Carlo simulation

» simulations estimate event likelihoods Random sampling

» generate a small random subproblem» solve, extrapolate to whole problem

Randomized Rounding for approximation

MIT

Cuts in Graphs

Focus on undirected graphs A cut is a vertex partition Value is number (or total weight) of

crossing edges

MIT

Optimization with Cuts

Cut values determine solution of many graph optimization problems:» min-cut / max-flow» multicommodity flow (sort-of)» bisection / separator» network reliability» network design

Randomization helps solve these problems

MIT

Presentation Assumption

For entire presentation, we consider unweighted graphs (all edges have weight/capacity one)

All results apply unchanged to arbitrarily weighted graphs» Integer weights = parallel edges» Rational weights scale to integers» Analysis unaffected» Some implementation details

MIT

Basic Probability

Conditional probability» Pr[A B] = Pr[A] Pr[B | A]

Independent events multiply:» Pr[A B] = Pr[A] Pr[B]

Linearity of expectation: » E[X + Y] = E[X] + E[Y]

Union Bound» Pr[X Y] Pr[X] + Pr[Y]

MIT

Random Selection forMinimum Cuts

Random choices are good

when problems are rare

MIT

Minimum Cut

Smallest cut of graph Cheapest way to separate into 2 parts Various applications:

» network reliability (small cuts are weakest)» subtour elimination constraints for TSP» separation oracle for network design

Not s-t min-cut

MIT

Max-flow/Min-cut

s-t flow: edge-disjoint packing of s-t paths s-t cut: a cut separating s and t [FF]: s-t max-flow = s-t min-cut

» max-flow saturates all s-t min-cuts» most efficient way to find s-t min-cuts

[GH]: min-cut is “all-pairs” s-t min-cut» find using n flow computations

MIT

Flow Algorithms

Push-relabel [GT]:» push “excess” around graph till it’s gone» max-flow in O*(mn) (note: O* hides logs)

– Recent O*(m3/2) [GR]

» min-cut in O*(mn2) --- “harder” than flow Pipelining [HO]:

» save push/relabel data between flows» min-cut in O*(mn) --- “as easy” as flow

MIT

Contraction

Find edge that doesn’t cross min-cut Contract (merge) endpoints to 1 vertex

MIT

Contraction Algorithm

Repeat n - 2 times:» find non-min-cut edge» contract it (keep parallel edges)

Each contraction decrements #vertices At end, 2 vertices left

» unique cut» corresponds to min-cut of starting graph

MIT

Picking an Edge

Must contract non-min-cut edges [NI]: O(m) time algorithm to pick edge

» n contractions: O(mn) time for min-cut» slightly faster than flows

If only could find edge faster….

Idea: min-cut edges are few

MIT

Randomize

Repeat until 2 vertices remainpick a random edge

contract it

(keep fingers crossed)

MIT

Analysis I

Min-cut is small---few edges» Suppose graph has min-cut c» Then minimum degree at least c» Thus at least nc/2 edges

Random edge is probably safe

Pr[min-cut edge] c/(nc/2)

= 2/n

(easy generalization to capacitated case)

MIT

Analysis II

Algorithm succeeds if never accidentally contracts min-cut edge

Contracts #vertices from n down to 2 When k vertices, chance of error is 2/k

» thus, chance of being right is 1-2/k Pr[always right] is product of

probabilities of being right each time

MIT

22

)1(2

31

132

32

122

n

2k

)())((

)1()1)(1(

safe]n contractio Pr[ success]Pr[

n

nn

nn

nn

nn

thk

Analysis III

…not too good!

MIT

Repetition

Repetition amplifies success probability» basic failure probability 1 - 2/n2

» so repeat 7n2 times

6

72

7

2

10

)1(

once]) (Pr[fail

times]7 Pr[failfailure] completePr[

2

2

2

n

n

n

n

MIT

How fast?

Easy to perform 1 trial in O(m) time» just use array of edges, no data structures

But need n2 trials: O(mn2) time Simpler than flows, but slower

MIT

An improvement [KS]

When k vertices, error probability 2/k» big when k small

Idea: once k small, change algorithm» algorithm needs to be safer» but can afford to be slower

Amplify by repetition!» Repeat base algorithm many times

MIT

Recursive Algorithm

Algorithm RCA ( G, n )

{G has n vertices}

repeat twice

randomly contract G to n/21/2 vertices

RCA(G,n/21/2)

(50-50 chance of avoiding min-cut)

MIT

Main Theorem

On any capacitated, undirected graph, Algorithm RCA» runs in O*(n2) time with simple structures» finds min-cut with probability 1/log n

Thus, O(log n) repetitions suffice to find the minimum cut (failure probability 10-6) in O(n2 log2 n) time.

MIT

Proof Outline

Graph has O(n2) (capacitated) edges So O(n2) work to contract, then two

subproblems of size n/2½

» T(n) = 2 T(n/2½) + O(n2) = O(n2 log n) Algorithm fails if both iterations fail

» Iteration succeeds if contractions and recursion succeed

» P(n)=1 - [1 - ½ P(n/2½)]2 = (1 / log n)

MIT

Failure Modes

Monte Carlo algorithms always run fast and probably give you the right answer

Las Vegas algorithms probably run fast and always give you the right answer

To make a Monte Carlo algorithm Las Vegas, need a way to check answer» repeat till answer is right

No fast min-cut check known (flow slow!)

MIT

How do we verify a minimum cut?

MIT

Enumerating Cuts

The probabilistic method, backwards

MIT

Cut Counting

Original CA finds any given min-cut with probability at least 2/n(n-1)

Only one cut found Disjoint events, so probabilities add So at most n(n-1)/2 min-cuts

» probabilities would sum to more than one Tight

» Cycle has exactly this many min-cuts

MIT

Enumeration

RCA as stated has constant probability of finding any given min-cut

If run O(log n) times, probability of missing a min-cut drops to 1/n3

But only n2 min-cuts So, probability miss any at most 1/n So, with probability 1-1/n, find all

» O(n2 log3 n) time

MIT

Generalization

If G has min-cut c, cut c is -mincut Lemma: contraction algorithm finds any

given -mincut with probability (n-2) » Proof: just add factor to basic analysis

Corollary: O(n2) -mincuts Corollary: Can find all in O*(n2) time

» Just change contraction factor in RCA

MIT

Summary

A simple fast min-cut algorithm» Random selection avoids rare problems

Generalization to near-minimum cuts Bound on number of small cuts

» Probabilistic method, backwards

MIT

Network Reliability

Monte Carlo estimation

MIT

The Problem

Input:» Graph G with n vertices » Edge failure probabilities

– For simplicity, fix a single p

Output:» FAIL(p): probability G is disconnected by

edge failures

MIT

Approximation Algorithms

Computing FAIL(p) is #P complete [V] Exact algorithm seems unlikely Approximation scheme

» Given G, p, , outputs -approximation» May be randomized:

– succeed with high probability

» Fully polynomial (FPRAS) if runtime is polynomial in n, 1/

MIT

Monte Carlo Simulation

Flip a coin for each edge, test graph k failures in t trials FAIL(p) k/t E[k/t] = FAIL(p) How many trials needed for confidence?

» “bad luck” on trials can yield bad estimate» clearly need at least 1/FAIL(p)

Chernoff bound: O*(1/2FAIL(p)) suffice to give probable accuracy within » Time O*(m/2FAIL(p))

MIT

Chernoff Bound

Random variables Xi [0,1] Sum X = Xi

Bound deviation from expectation

Pr[ |X-E[X]| E[X] ] < exp(-2E[X]/4) If E[X] 4(log n)/2, “tight concentration”

» Deviation by probability < 1 / n No one variable is a big part of E[X]

MIT

Application

Let Xi=1 if trial i is a failure, else 0

Let X = X1 + … + Xt

Then E[X] = t FAIL(p) Chernoff says X within relative of E[X]

with probability 1-exp(2 t FAIL(p)/4) So choose t to cancel other terms

» “High probability” t = O(log n / 2FAIL(p))» Deviation by probability < 1 / n

MIT

Review

Contraction Algorithm» O(n2) -mincuts» Enumerate in O*(n2) time

MIT

Network reliability problem

Random edge failures» Estimate FAIL(p) = Pr[graph disconnects]

Naïve Monte Carlo simulation» Chernoff bound---“tight concentration”

Pr[ |X-E[X]| E[X] ] < exp(-2E[X]/4)» O(log n / 2FAIL(p)) trials expect O(log n / 2)

network failures---good for Chernoff» So estimate within in O*(m/2FAIL(p)) time

MIT

Rare Events

When FAIL(p) too small, takes too long to collect sufficient statistics

Solution: skew trials to make interesting event more likely

But in a way that let’s you recover original probability

MIT

DNF Counting

Given DNF formula (OR of ANDs)(e1 e2 e3) (e1 e4) (e2 e6)

Each variable set true with probability p Estimate Pr[formula true]

» #P-complete [KL, KLM] FPRAS

» Skew to make true outcomes “common”» Time linear in formula size

MIT

Rewrite problem

Assume p=1/2 » Count satisfying assignments

“Satisfaction matrix » Sij=1 if ith assignment satisfies jth clause

We want number of nonzero rows Randomly sampling rows won’t work

» Might be too few nonzeros

MIT

New sample space

So normalize every nonzero row to sum to one (divide by number of nonzeros)» Now sum of nonzeros is desired value» So sufficient to estimate average nonzero

MIT

Sampling Nonzeros

We know number of nonzeros/column» If satisfy given clause, all variables in

clause must be true» All other variables unconstrained

Estimate average by random sampling» Know number of nonzeros/column» So can pick random column» Then pick random true-for-column

assignment

MIT

Few Samples Needed

Suppose k clauses Then E[sample] > 1/k

» 1 satisfied clauses k» 1 sample value 1/k

Adding O(k log n / 2) samples gives “large” mean

So Chernoff says sample mean is probably good estimate

MIT

Reliability Connection

Reliability as DNF counting:» Variable per edge, true if edge fails» Cut fails if all edges do (AND of edge vars)» Graph fails if some cut does (OR of cuts)» FAIL(p)=Pr[formula true]

Problem: the DNF has 2n clauses

MIT

Focus on Small Cuts

Fact: FAIL(p) > pc

Theorem: if pc=1/n(2+) then Pr[>-mincut fails]< n-

Corollary: FAIL(p) Pr[-mincut fails],

where=1+2/ Recall: O(n2) -mincuts Enumerate with RCA, run DNF counting

MIT

Proof of Theorem

Given pc=1/n(2+)

At most n2 cuts have value c Each fails with probability pc=1/n(2+)

Pr[any cut of value c fails] = O(n) Sum over all

MIT

Algorithm

RCA can enumerate all -minimum cuts with high probability in O(n2) time.

Given -minimum cuts, can -estimate probability one fails via Monte Carlo simulation for DNF-counting (formula size O(n2))

Corollary: when FAIL(p)< n-(2+), can -approximate it in O (cn2+4/) time

MIT

Combine

For large FAIL(p), naïve Monte Carlo For small FAIL(p), RCA/DNF counting Balance: -approx. in O(mn3.5/2) time Implementations show practical for

hundreds of nodes Again, no way to verify correct

MIT

Summary

Naïve Monte Carlo simulation works well for common events

Need to adapt for rare events Cut structure and DNF counting lets us

do this for network reliability

MIT

Random Sampling

More min-cut algorithms

MIT

Random Sampling

General tool for faster algorithms:» pick a small, representative sample» analyze it quickly (small)» extrapolate to original (representative)

Speed-accuracy tradeoff» smaller sample means less time» but also less accuracy

MIT

Min-cut Duality

[Edmonds]: min-cut=max tree packing» convert to directed graph» “source” vertex s (doesn’t matter which)» spanning trees directed away from s

[Gabow] “augmenting trees”» add a tree in O*(m) time» min-cut c (via max packing) in O*(mc)

» great if m and c are small…

MIT

Example

min-cut 2

2 directedspanning trees

directed min-cut 2

MIT

Sampling for Approximation

MIT

Random Sampling

[Gabow] scheme great if m, c small Random sampling

» reduces m, c

» scales cut values (in expectation)» if pick half the edges, get half of each cut

So find tree packings, cuts in samples

Problem: maybe some large deviations

MIT

Sampling Theorem

Given graph G, build a sample G(p) by including each edge with probability p

Cut of value v in G has expected value pv in G(p)

Definition: “constant” = 8 (ln n) / 2

Theorem: With high probability, all cuts in G( / c) have (1 ± ) times their expected values.

MIT

A Simple Application

[Gabow] packs trees in O*(mc) time Build G( / c)

» minimum expected cut » by theorem, min-cut probably near » find min-cut in time O*(m) using [Gabow]» corresponds to near-min-cut in G

Result: (1+) times min-cut in O*(m) time

MIT

Proof of Sampling: Idea

Chernoff bound says probability of large deviation in cut value is small

Problem: exponentially many cuts. Perhaps some deviate a great deal

Solution: showed few small cuts» only small cuts likely to deviate much» but few, so Chernoff bound applies

MIT

Proof of Sampling

Sampled with probability /c, » a cut of value c has mean » [Chernoff]: deviates from expected size by

more than with probability at most n-3

At most n2 cuts have value c Pr[any cut of value c deviates] = O(n) Sum over all

MIT

Las Vegas Algorithms

Finding Good Certificates

MIT

Approximate Tree Packing

Break edges into c /random groups Each looks like a sample at rate / c

» O*( m / c) edges» each has min expected cut » so theorem says min-cut 1 –

So each has a packing of size 1 – [Gabow] finds in time O*(m/c) per group

» so overall time is c O*(m/c) = O*(m)

MIT

Las Vegas Algorithm

Packing algorithm is Monte Carlo Previously found approximate cut (faster) If close, each “certifies” other

» Cut exceeds optimum cut» Packing below optimum cut

If not, re-run both Result: Las Vegas, expected time O*(m)

MIT

Exact Algorithm

Randomly partition edges in two groups» each like a 1/2-sample: =O*(c-1/2)

Recursively pack trees in each half» c/2 - O*(c1/2) trees

Merge packings» gives packing of size c - O*(c1/2)

» augment to maximum packing: O*(mc1/2) T(m,c)=2T(m/2,c/2)+O*(mc1/2) = O* (mc1/2)

MIT

Nearly Linear Time

MIT

Analyze Trees

Recall: [G] packs c (directed)-edge disjoint spanning trees

Corollary: in such a packing, some tree crosses min-cut only twice

To find min-cut:» find tree packing» find smallest cut with 2 tree edges crossing

Problem: packing takes O*(mc) time

MIT

Constraint trees

Min-cut c: » c directed trees» 2c directed min-cut

edges » On average, two

min-cut edges/tree

Definitions:

tree 2-crosses cut

MIT

Finding the Cut

From crossing tree edges, deduce cut

Remove tree edges

No other edges cross So each component

is on one side And opposite its

“neighbor’s” side

MIT

Sampling

Solution: use G(/c) with =1/8» pack O*() trees in O*(m) time » original min-cut has (1) edges in G( / c) » some tree 2-crosses it in G( / c) » …and thus 2-crosses it in G

Analyze O*() trees in G» time O*(m) per tree» Monte Carlo

MIT

Simplify

Discuss case where one tree edge crosses min-cut

MIT

Root tree, so cut subtree Use dynamic program up from leaves to

determine subtree cuts efficiently Given cuts at children of a node,

compute cut at parent Definitions:

» v are nodes below v» C(v) is value of cut at subtree v

Analyzing a tree

MIT

The Dynamic Program

u

v w

keep

discard

C u C v C w C v w 2 ,

Edges with leastcommon ancestor u

MIT

Algorithm: 1-Crossing Trees

Compute edges’ LCA’s: O(m) Compute “cuts” at leaves

» Cut values = degrees» each edge incident on at most two leaves» total time O(m)

Dynamic program upwards: O(n)

Total: O(m+n)

MIT

2-Crossing Trees

Cut corresponds to two subtrees:

n2 table entries fill in O(n2) time with dynamic program

C v w C v C w C v w 2 ,

v w

keep

discard

MIT

Bottleneck is C(v, w) computations Avoid. Find right “twin” w for each v

Linear Time

Compute using addpath and minpath operations of dynamic trees [ST]

Result: O(m log3 n) time (messy)

C v C w C v ww

min ,2

MIT

How do we verify a minimum cut?

MIT

Network Design

Randomized Rounding

MIT

Problem Statement

Given vertices, and cost cvw to buy and edge from v to w, find minimum cost purchase that creates a graph with desired connectivity properties

Example: minimum cost k-connected graph.

Generally NP-hard Recent approximation algorithms [GW],

[JV]

MIT

Integer Linear Program

Variable xvw=1 if buy edge vw

Solution cost xvw cvw

Constraint: for every cut, xvw k

Relaxing integrality gives tractable LP» Exponentially many cuts» But separation oracles exist (eg min-cut)

What is integrality gap?

MIT

Randomized Rounding

Given LP solution values xvw

Build graph where vw is present with probability xvw

Expected cost is at most opt: xvw cvw

Expected number of edges crossing any cut satisfies constraint

If expected number large for every cut, sampling theorem applies

MIT

k-connected subgraph

Fractional solution is k-connected So every cut has (expected) k edges

crossing in rounded solution Sampling theorem says every cut has at

least k-(k log n)1/2 edges Close approximation for large k Can often repair: e.g., get k-connected

subgraph at cost 1+((log n)/k)1/2 times min

MIT

Nonuniform Sampling

Concentrate on the important things

[Benczur-Karger, Karger, Karger-Levine]

MIT

s-t Min-Cuts

Recall: if G has min-cut c, then in G(/c) all cuts approximate their expected values to within.

Applications:

Min-cut inO*(mc) time [G]

Approximate/exact inO*((m/c) c) =O*(m)

s-t min-cut of value v in O*(mv)

Approximate inO*(mv/c) time

Trouble if c is small and v large.

MIT

The Problem

Cut sampling relied on Chernoff bound Chernoff bounds required that no one

edge is a large fraction of the expectation of a cut it crosses

If sample rate 1/c, each edge across a min-cut is too significant

But: if edge only crosses large cuts, then sample rate 1/c is OK!

MIT

Biased Sampling

Original sampling theorem weak when» large m

» small c But if m is large

» then G has dense regions» where c must be large» where we can sample more sparsely

MIT

Approx. s-t min-cut O*(mn) O*(n2 / 2)

Approx. s-t max-flow O*(m3/2 ) O*(mn1/2 / )

Flow of value v O*(mv)

O*(n11/9v)

Approx. bisection O*(m2) O*(n2 / 2)

m n /2 in weighted, undirected graphs

Problem Old Time New Time

vmnO *

MIT

Definition: A k-strong component is a maximal vertex-induced subgraph with min-cut k.

Strong Components

2 2

3

3

MIT

Nonuniform Sampling

Definition: An edge is k-strong if its endpoints are in same k-component.

Stricter than k-connected endpoints. Definition: The strong connectivity ce

for edge e is the largest k for which e is k-strong.

Plan: sample dense regions lightly

MIT

Nonuniform Sampling

Idea: if an edge is k-strong, then it is in a k-connected graph

So “safe” to sample with probability 1/k Problem: if sample edges with different

probabilities, E[cut value] gets messy Solution: if sample e with probability pe,

give it weight 1/pe

Then E[cut value]=original cut value

MIT

Compression Theorem

Definition: Given compression probabilities pe, compressed graph G[pe]

» includes edge e with probability pe and

» gives it weight 1/pe if included

Note E[G[pe]] = G

Theorem: G[/ ce]

» approximates all cuts by » has O (n) edges

MIT

Proof (approximation)

Basic idea: in a k-strong component, edges get sampled with prob. / k» original sampling theorem works

Problem: some edges may be in stronger components, sampled less

Induct up from strongest components:» apply original sampling theorem inside» then “freeze” so don’t affect weaker parts

MIT

Strength Lemma

Lemma: 1/ce n

» Consider connected component C of G» Suppose C has min-cut k

» Then every edge e in C has ce k

» So k edges crossing C’s min-cut have

1/ce 1/k k (1/k ) = 1

» Delete these edges (“cost” 1)» Repeat n - 1 times: no more edges!

MIT

Proof (edge count)

Edge e included with probability / ce

So expected number is / ce

We saw 1/ce n

So expected number at most n

MIT

Construction

To sample, must find edge strengths» can’t, but approximation suffices

Sparse certificates identify weak edges:» construct in linear time [NI]» contain all edges crossing cuts k

» iterate until strong components emerge Iterate for 2i-strong edges, all i

» tricks turn it strongly polynomial

MIT

Certificate Algorithm

Repeat k times» Find a spanning forest» Delete it

Each iteration deletes one edge from every cut (forest is spanning)

So at end, any edge crossing a cut of size k is deleted

[NI] merge all iterations in O(m) time

MIT

Flows

Uniform sampling led to flow algorithms» Randomly partition edges» Merge flows from each partition element

Compression problematic» Edge capacities changed» So flow path capacities distorted» Flow in compressed graph doesn’t fit in

original graph

MIT

Smoothing

If edge has strength ce, divide into / ce edges of capacity ce /» Creates 1/ce = n edges

Now each edge is only 1/ fraction of any cut of its strong component

So sampling a 1/ fraction works So dividing into groups works Yields (1-) max-flow in time /* vmnO

MIT

Cleanup

Approximate max-flow can be made exact by augmenting paths

Integrality problems» Augmenting paths fast for small integer flow» But breakup by smoothing ruins integrality

Surmountable» Flows in dense and sparse parts separable

Result: max-flow in O*(n11/9v) time

MIT

Proof By Picture

s t

Dense regions

MIT

Compress Dense Regions

s t

MIT

Solve Sparse Flow

s t

MIT

Replace Dense Parts(keep flow in sparse bits)

s t

MIT

“Fill In” Dense Parts

s t

MIT

Conclusions

MIT

Conclusion

Randomization is a crucial tool for algorithm design

Often yields algorithms that are faster or simpler than traditional counterparts

In particular, gives significant improvements for core problems in graph algorithms

MIT

Randomized Methods

Random selection» if most candidate choices “good”, then a

random choice is probably good Monte Carlo simulation

» simulations estimate event likelihoods Random sampling

» generate a small random subproblem» solve, extrapolate to whole problem

Randomized Rounding for approximation

MIT

Random Selection

When most choices good, do one at random

Recursive contraction algorithm for minimum cuts» Extremely simple (also to implement)» Fast in theory and in practice [CGKLS]

MIT

Monte Carlo

To estimate event likelihood, run trials Slow for very rare events Bias samples to reveal rare event FPRAS for network reliability

MIT

Random Sampling

Generate representative subproblem Use it to estimate solution to whole

» Gives approximate solution» May be quickly repaired to exact solution

Bias sample toward “important” or “sensitive” parts of problem

New max-flow and min-cut algorithms

MIT

Randomized Rounding

Convert fractional to integral solutions Get approximation algorithms for integer

programs “Sampling” from a well designed sample

space of feasible solutions Good approximations for network

design.

MIT

Generalization

Our techniques work because undirected graph are matroids

All our results extend/are special cases» Packing bases» Finding minimum “quotients”» Matroid optimization (MST)

MIT

Directed Graphs?

Directed graphs are not matroids Directed graphs can have lots of

minimum cuts Sampling doesn’t appear to work Residual graphs for flows are directed

» Precludes obvious recursive solutions to flow problems

MIT

Open problems

Flow in O(nv) time (complete m n)» Eliminate v dependence» Apply to weighted graphs with large flows» Flow in O(m) time?

Las Vegas algorithms» Finding good certificates

Detrministic algorithms» Deterministic construction of “samples”» Deterministically compress a graph

MIT

Randomization in Graph Optimization

ProblemsDavid Karger

MIT

http://theory.lcs.mit.edu/~karger

karger@mit.edu