Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.
-
Upload
hailey-tennies -
Category
Documents
-
view
219 -
download
1
Transcript of Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.
![Page 1: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/1.jpg)
Primal Dual Primal Dual Combinatorial Combinatorial
AlgorithmsAlgorithms
Qihui ZhuQihui ZhuMay 11, 2009May 11, 2009
![Page 2: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/2.jpg)
OutlineOutline
• Packing and covering
• Primal and dual problems
• Online prediction using feedback
• Primal-dual combinatorial
algorithm
• Applications and extensions
![Page 3: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/3.jpg)
Packing Packing Problem (0-1 Knapsack)Problem (0-1 Knapsack)
• n objects with weights Wi and prices pi
• A knapsack with capacity c
• Pack objects maximizing the total price
without exceeding the capacity
![Page 4: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/4.jpg)
Packing Problem (0-1 Knapsack)Packing Problem (0-1 Knapsack)
• n objects with weights Wi and prices pi
• A knapsack with capacity c
• Pack objects maximizing the total price
without exceeding the capacity
Knapsack IP
![Page 5: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/5.jpg)
Packing Problem (General)Packing Problem (General)
• n objects with weights Wij and prices pi
• m constraints with capacity cj to fit
• Pack objects maximizing the total price
without exceeding the capacities
Packing IP
![Page 6: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/6.jpg)
Covering ProblemCovering Problem
• n objects to cover at least pi times
• m sets with costs cj cover
each covers Wji objects
• Cover all objects with the minimal cost
![Page 7: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/7.jpg)
Covering ProblemCovering Problem
• n objects to cover at least pi times
• m sets with costs cj cover
each covers Wji objects
• Cover all objects with the minimal cost
Covering IP
![Page 8: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/8.jpg)
Fractional Packing Fractional Packing
Original form
![Page 9: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/9.jpg)
Fractional Packing Fractional Packing
Matrix form
Original form
![Page 10: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/10.jpg)
Fractional Packing and Covering Fractional Packing and Covering Are Dual Are Dual
Packing Covering
![Page 11: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/11.jpg)
Fractional Packing and Covering Fractional Packing and Covering Are Dual Are Dual
Packing Covering
Covering: best upper bound on packingPacking: best lower bound on covering
![Page 12: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/12.jpg)
Decision VersionDecision Version
Given constraint set
such that
For packing:
(PST 95)
Feasibility
Binary search for optimization
![Page 13: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/13.jpg)
Flipped Sides of the Same CoinFlipped Sides of the Same Coin
“Yes” certificate “No” certificate
Think of the game of twenty questions...
![Page 14: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/14.jpg)
Key Idea #1: Need Primal-Dual Key Idea #1: Need Primal-Dual AlgorithmAlgorithm
“Yes” certificate “No” certificate Primal: Dual:
![Page 15: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/15.jpg)
How to Generate the How to Generate the Certificates?Certificates?
Randomly guessing ...
![Page 16: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/16.jpg)
How to Generate the How to Generate the Certificates?Certificates?
Randomly guessing ...
Primal and Dual need to communicate
![Page 17: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/17.jpg)
From Dual to PrimalFrom Dual to Primal
Efficient combinatorial algorithm
![Page 18: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/18.jpg)
From Dual to PrimalFrom Dual to Primal
Given dual estimate and constraint set
Let , find such that
Oracle
Efficient combinatorial algorithm
![Page 19: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/19.jpg)
From Dual to PrimalFrom Dual to Primal
Given dual estimate and constraint set
Let , find such that
Oracle
For packing, essentially ignore all capacity constraints
Reduce to sorting over !Complexity:
![Page 20: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/20.jpg)
From Primal to DualFrom Primal to Dual
Combination of hyperplanes
![Page 21: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/21.jpg)
From Primal to DualFrom Primal to Dual
Combination of hyperplanes
Tilt the hyperplanes using feedback , ..., etc.
One step not enough!
Getting complicated over iterations...
Online Prediction
![Page 22: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/22.jpg)
Online PredictionOnline Prediction
• Experts predicting some uncertain event
Expert
Event
![Page 23: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/23.jpg)
Online PredictionOnline Prediction
• Experts predicting some uncertain event
Expert
Event
Value
• Gain some value from the world (adversarial)
![Page 24: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/24.jpg)
Online PredictionOnline Prediction
• Experts predicting some uncertain event
Expert
Event
Value
• Gain some value from the world (adversarial)
Weight
• Linearly combine by weights
![Page 25: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/25.jpg)
Online PredictionOnline Prediction
• Experts predicting some uncertain event
Expert
Event
• Long term value over time
Time
Value
• Gain some value from the world (adversarial)
Weight
• Linearly combine by weights
![Page 26: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/26.jpg)
A Simplified CaseA Simplified Case
Expert 1
Expert 2
Combined
Value Average playoff
Regret
1 0 1 0 1 0 1
history
? ? ? ? ?
• Only 2 experts
0 1 0 1 0 1 0? ? ? ? ?
?
?
• Value
![Page 27: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/27.jpg)
0.9 0 0 0 0 0 0
Strategy IStrategy I
Expert 1
Expert 2
Combined
0.9 0 1 0 1 0 11 0 1 0 1
Take the best from history?
0.1 1 0 1 0 1 01 0 1 0 1
1
0
1
0 10.5
![Page 28: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/28.jpg)
0 1 0 2/32/31/32/3
Strategy IIStrategy II
Expert 1
Expert 2
Combined
1 1 0 1 1 0 1
Linearly weighted by the cumulative values?
0 0 1 0 0 1 0
0
1
0 10.5
1 1 2/3 2/3 2/3
1 0 0 1/3 1/3 1/3
![Page 29: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/29.jpg)
0 1 0 1 1 0 1
Strategy IIIStrategy III
Expert 1
Expert 2
Combined
1 1 0 1 1 0 1
Exponentially weighted by the cumulative values!
0 0 1 0 0 1 0
0
1
0 10.5
1 1 1-ε
1 0 0 ε ε ε
1-ε 1-ε
![Page 30: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/30.jpg)
Key Idea #2: Need Multiplicative Key Idea #2: Need Multiplicative FeedbackFeedback
“Yes” certificate “No” certificate Primal: Dual:
![Page 31: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/31.jpg)
From Primal to Dual: ContinuedFrom Primal to Dual: Continued
Multiplicative Weight Update (MW)
Let be the current state at step and be the “feedback”
Combination of hyperplanes
![Page 32: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/32.jpg)
Regret BoundRegret Bound
Theorem (LW94) Suppose predictions of experts
have value . Each time the predictions are
combined by weights , where .
Update weights using . After time
![Page 33: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/33.jpg)
Regret BoundRegret Bound
Theorem (LW94) Suppose predictions of experts
have value . Each time the predictions are
combined by weights , where .
Update weights using . After time
Proof. Consider the potential function
Show that it is dominated by the best expert.
![Page 34: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/34.jpg)
Primal-Dual AlgorithmsPrimal-Dual Algorithms
Primal Dual
Multiplicative Weight Update
Oracle
![Page 35: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/35.jpg)
AlgorithmAlgorithm
Initialize: ,
Repeat
Set
Oracle
If then
Return infeasible, output
Set
Multiplicative Weight Update
Until
Return feasible solution
Primal-Dual Combinatorial Algorithm
![Page 36: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/36.jpg)
AnalysisAnalysis
Initialize: ,
Repeat
Set
Oracle
If then
Return infeasible, output
Set
Multiplicative Weight Update
Until
Return feasible solution
Primal-Dual Combinatorial AlgorithmOracle: if failed, infeasible
![Page 37: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/37.jpg)
AnalysisAnalysis
Initialize: ,
Repeat
Set
Oracle
If then
Return infeasible, output
Set
Multiplicative Weight Update
Until
Return feasible solution
Primal-Dual Combinatorial Algorithm Oracle: if failed, infeasible
Regret bound:
Width:
Feasible solution!
Complexity depends on
![Page 38: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/38.jpg)
ApplicationsApplications
• Fractional packing and covering
• Multicommodity flow
• Held-Karp Bound for TSP
• Semidefinite programming (SDP)
relaxation
• General convex programming
• Boosting
• Matrix game
![Page 39: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/39.jpg)
ApplicationsApplications
• Fractional packing and covering
• Multicommodity flow
• Held-Karp Bound for TSP
• Semidefinite programming (SDP)
relaxation
• General convex programming
• Boosting
• Matrix game
![Page 40: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/40.jpg)
Multicommodity FlowMulticommodity Flow
• Objective: maximizing total flow while respecting capacities
![Page 41: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/41.jpg)
Multicommodity FlowMulticommodity Flow
• Objective: maximizing total flow while respecting capacities
• Packing paths
• Update: route some flow along shortest path
Multicommodity Flow
Congestion
![Page 42: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/42.jpg)
Initialize: ,
Repeat
Set
Oracle
If then
Return infeasible, output
Set
Multiplicative Weight Update
Until
Return feasible solution
Primal-Dual Combinatorial Algorithm
Multicommodity FlowMulticommodity Flow
Oracle : shortest path
keep pushing flows through
P : polytope of graph flows
Augment length (GK 98)
![Page 43: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/43.jpg)
SDP RelaxationSDP RelaxationMax Cut (GW 95)
• Better bounds on Max Cut, Sparsest Cut, Max-2Sat, etc.
• Finding good geometric embedding by SDP + rounding
• Interior point method not scale-
Sparsest Cut (ARV 04)
Primal SDP Dual SDP
![Page 44: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/44.jpg)
Initialize: ,
Repeat
Set
Oracle
If then
Return infeasible, output
Set
Multiplicative Weight Update
Until
Return feasible solution
Primal-Dual Combinatorial Algorithm
Primal SDPPrimal SDP
P : set of semidefinite matrices
Oracle : compute eigenvectors!
![Page 45: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/45.jpg)
Initialize: ,
Repeat
Set
Oracle
If then
Return infeasible, output
Set
Multiplicative Weight Update
Until
Return feasible solution
Primal-Dual Combinatorial Algorithm
Dual SDPDual SDP
P & Oracle : same as packing
Multiplicative Weight Update
Use matrix exponential
!
![Page 46: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/46.jpg)
Initialize: ,
Repeat
Set
Oracle
If then
Return infeasible, output
Set
Multiplicative Weight Update
Until
Return feasible solution
Primal-Dual Combinatorial Algorithm
Weighted Majority Weighted Majority Algorithm/BoostingAlgorithm/Boosting
Events/Classification results: not controlled by us!
Error on training examples
![Page 47: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/47.jpg)
SummarySummary
Combinatorial subroutines
![Page 48: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/48.jpg)
SummarySummary
Combinatorial subroutines
Fast algorithms!
![Page 49: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/49.jpg)
ConclusionConclusion
• A computational paradigm– Applied to numerous problems – Cover at least convex problems
• Fast approximation algorithms– Trade accuracy for time: O(1/ε)– Fast combinatorial algorithms for oracle (without
solving LP!)
• Flexibility for algorithm design– Multiplicative weight update: a principle way to
use feedback– Free to separate and combine constraints
• Width management is important– Most algorithms polynomial to width– Separate high width constraints: nested primal-
dual– Decompose constraint set
![Page 50: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/50.jpg)
Thank you!Thank you!
Questions …Questions …
![Page 51: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/51.jpg)
Regret BoundRegret Bound
Theorem (LW94) Suppose predictions of experts
have value . Each time the predictions are
combined by weights , where .
Update weights using . After time
Proof. Consider the potential function
Show that it is dominated by the best expert.
![Page 52: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/52.jpg)
Update rule
![Page 53: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/53.jpg)
Update rule
Exp function
Width
![Page 54: Primal Dual Combinatorial Algorithms Qihui Zhu May 11, 2009.](https://reader036.fdocuments.net/reader036/viewer/2022062417/551a5102550346cb358b5c07/html5/thumbnails/54.jpg)
Update rule
Exp function
Width