Duality in Optimization and Constraint Satisfaction
J. N. HookerCarnegie Mellon Univ.
Pittsburgh, USA
September 2006
Some Concepts of Duality
• Reflexive dual (mirror image)– Conservation of
parity• Refuted by C. S.
Wu, 1957
Some Concepts of Duality
• Dual perspectives on one reality– Wave/particle duality– Mind/body duality
• Spinoza, Brahman/Atman
Lalita Mahatripurasundarirepresenting Brahman-Atman
consciousness
Some Concepts of Duality
• Complementarity– Yin/yang– Good/evil in
Zoroastrianism
Aharamazda, Zoroastrian god
Duality in Mathematics
• Duality of polytopes– k-dimensional faces become (n k 1)-dimensional
faces
Duality in Mathematics
• Duality of points and lines in projective geometry
7-point projective plane(Fano plane)
Duality in Mathematics
• Duality of rows and columns in matrix algebra.– Row rank of a matrix = column rank.
Duality in Mathematics
• Duality of rows and columns in matrix algebra.– Row rank of a matrix = column rank.
• Generalization: dual space of a vector space.– Space of linear functionals on a space has same
dimension as original space.– Dual of dual is a “natural transformation” in category
theory.
Duality in Optimization
• Most optimization duals offer dual perspectives.– Not reflexive in general.
Duality in Optimization
• Reflexive:– Linear programming dual
• Not reflexive:– Surrogate dual– Lagrangean dual– Superadditive dual
Duality in Optimization
• Reflexive:– Linear programming dual
• Not reflexive:– Surrogate dual– Lagrangean dual– Superadditive dual
• All of these are instances of inference duality and relaxation duality.
Duality in Optimization
• Inference duality and relaxation duality are very different concepts.– Even though several optimization duals can be
viewed as either.
Duality in Optimization
• Inference duality and relaxation duality are very different concepts.– Even though several optimization duals can be
viewed as either.• Inference duality provides sensitivity analysis and
nogood-based search.– Will focus on nogood-based search here.
Duality in Optimization
• Inference duality and relaxation duality are very different concepts.– Even though several optimization duals can be
viewed as either.• Inference duality provides sensitivity analysis and
nogood-based search.– Will focus on nogood-based search here.
• Relaxation duality provides tight bounds.
Duality in Optimization
• Inference duality and relaxation duality are very different concepts.– Even though several optimization duals can be
viewed as either.• Inference duality provides sensitivity analysis and
nogood-based search.– Will focus on nogood-based search here.
• Relaxation duality provides tight bounds.• The constraint dual is neither type but gives rise to a
relaxation dual.
Duality in Optimization
• Inference duality is a duality of search and inference.– The original problem can be solved by searching
over values of variables to find the best solution.– The dual is solved by inferring from the constraints
the best bound on the optimal value.
Duality in Optimization
• Inference duality is a duality of search and inference.– The original problem can be solved by searching
over values of variables to find the best solution.– The dual is solved by inferring from the constraints
the best bound on the optimal value.
• Relaxation duality is a duality of restriction and relaxation.– The original problem can be solved by enumerating
restrictions.– The dual can be solved by enumerating
parameterized relaxations.
Duality in Optimization
• Viewing duals in this way has three benefits.• It reveals connections that might not otherwise be
noticed.– For instance, surrogate and Lagrangean duals are
seen to be closely related when viewed as inference duals rather than as relaxation duals.
Duality in Optimization
• Viewing duals in this way has three benefits.• It reveals connections that might not otherwise be
noticed.– For instance, surrogate and Lagrangean duals are
seen to be closely related when viewed as inference duals rather than as relaxation duals.
• It can unify solution methods.– For instance, Benders decomposition and DPL with
clause learning are nogood-based search methods that result from different inference duals.
Duality in Optimization
• It can suggest new solution methods.– For instance, any domain filter defines an inference
dual with an associated nogood-based search methods.
• Edge-finding gives rise to an effective Benders-like decomposition method for planning & scheduling, etc.
– A relaxation dual can be defined on a constraint dual formulation so as to generalize mini-bucket elimination.
Inference Dual
• An optimization problem minimizes an objective function subject to constraints.– It is solved by searching over values of the
variables.
• The inference dual finds the tightest lower bound on the objective function that can be deduced from the constraints.– It is solved by searching over proofs.
Inference Dual
• The optimization problem
has the inference dual
min ( )x D
f x C
max ( )
P Pv f x vC
P
Constraint set
Family of proofs
Proof P deduces f(x) v from C
“Primal” problem
Inference Dual
• Weak duality always holds:
Max value of dual problem
Min value of primal problem
Difference = duality gap
max ( )
P Pv f x vC
P
Inference Dual
• Strong duality sometimes holds:
Max value of dual problem
= Min value of primal problem
P is a complete proof family
Strong duality
max ( )
P Pv f x vC
P
Linear Programming: Inference Dual
• A linear programming problem has the form
• The inference dual is
– The proof family P consists of domination by surrogates (nonnegative linear combinations).
0
minx
cx Ax b
max
P Pv Ax b cx v
P
Linear Programming: Inference Dual
• For any u 0, uAx ub is a surrogate of Ax b.
• uAx ub dominates cx v if uAx ub implies cx v (for x 0).– That is, uA c and ub v.
• This is a complete inference method.– Due to Farkas Lemma.
Linear Programming: Inference Dual
• So the inference dual
becomes
or
– This is the classical linear programming dual.
max
P Pv Ax b cx v
P
0
max ,u
v uA c ub v
0
maxu
ub uA c
when Ax b, x 0 is feasible
Linear Programming: Inference Dual
• Since the proof family is complete, we have strong duality:
0 0
mimax nxu
ub A cxu Axc b
except when uA c, u 0 and Ax b, x 0 are both infeasible.
Linear Programming: Inference Dual
• Since the proof family is complete, we have strong duality:
– In this case, the dual is symmetric• The dual of the dual is the primal.
– The dual therefore belongs to NP and the primal to co-NP.
0 0
mimax nxu
ub A cxu Axc b
except when uA c, u 0 and Ax b, x 0 are both infeasible.
Linear Programming: Inference Dual
• Example:
1 2
1 2 1
1 2 2
1 2
min 4 7
2 3 6 ( )
2 4 ( )
, 0
x x
x x u
x x u
x x
1 2
1 2 1
1 2 2
1 2
max 6 4
2 2 4 ( )
3 7 ( )
, 0
u u
u u x
u u x
u u
Dual:
Linear Programming: Inference Dual
• Example:
– Solution:
Dual:
1 2
1 2
1
1
2
2
(2)
(0)
2 3 6
2
4
4
4
6
7
12
12
x x
x x
x
x
x
xSurrogateDominates
1 2
1 2 1
1 2 2
1 2
12
2
min 4 7
2 3 6 ( )
2 4 ( )
, 0
0
x x
x x u
x x u
x x
1 2
1 2 1
1 2 2
1 2
12
3
max 6 4
2 2 4 ( )
3 7 ( )
, 0
0
u u
u u x
u u x
u u
Relaxation Dual
• A relaxation dual parameterizes relaxations of the optimization problem.– The relaxation parameters are dual variables.– Each relaxation provides a lower bound on the
optimal value of the primal.
• It seeks the relaxation that provides the tightest bound.– It is solved by searching over values of the dual
variables.
Relaxation Dual
• Given the optimization problem
a parameterized relaxation is
min ( )x D
f x C
min ( , ) ( )x D
f x u uC
Lower bound on f(x) for all x D satisfying C.
Relaxation of C.
u = vector of dual variables
Relaxation Dual
• Given the optimization problem
a parameterized relaxation is
– It provides a lower bound on the optimal value of the original problem, for any u.
min ( )x D
f x C
min ( , ) ( )x D
f x u uC
Relaxation Dual
• Given the optimization problem
a relaxation dual is
min ( )x D
f x C
max min ( , ) ( )
x Du Uf x u uC
Find relaxation that provides the largest lower bound.
Relaxation Dual
• Weak duality always holds:
Max value of dual problem
Min value of primal problem
Difference = duality gap
max min ( , ) ( )
x Du Uf x u uC
Linear Programming: Relaxation Dual
• A surrogate relaxation of
replaces Ax b with a surrogate uAx ub.– The relaxation is parameterized by u.– The objective function is left unchanged.
0
minx
cx Ax b
Linear Programming: Relaxation Dual
• This yields the relaxation dual
– It is equivalent to the classical linear programming dual.
– So the classical dual is both an inference and a relaxation dual.
00max min
xucx uAx ub
Linear Programming: Relaxation Dual
• Example:
• Relaxation dual:
1 21 21 2 1 2 1 1 2 2 1 2, 0, 0
max min 4 7 (2 2 ) (3 ) 6 4x xu u
x x u u x u u x u u
1 2
1 2 1
1 2 2
1 2
min 4 7
2 3 6 ( )
2 4 ( )
, 0
x x
x x u
x x u
x x
Linear Programming: Relaxation Dual
• Example:
• Relaxation dual:
• Solution:
1 21 21 2 1 2 1 1 2 2 1 2, 0, 0
max min 4 7 (2 2 ) (3 ) 6 4x xu u
x x u u x u u x u u
1 2
1 2 1 2, 0min 4 7 124 6 12x x
x x x x
1 2Let ( , ) (2,0)u u
1 2
1 2 1
1 2 2
1 2
min 4 7
2 3 6 ( )
2 4 ( )
, 0
x x
x x u
x x u
x x
Surrogate Dual
• The surrogate dual is a relaxation dual of a general inequality-constrained problem:
• The parameterized relaxation is
• So the dual is
– In general there is a duality gap.
min ( ) ( ) 0x D
f x g x
min ( ) ( ) 0x D
f x ug x
0max min ( ) ( ) 0
u x Df x ug x
Surrogate Dual
• Example:
• The surrogate dual is
1 2
1 2
1 2
1 2
min 3 4
3 0
5 2 0
, {0,1,2,3}
x x
x x
x x
x x
1 2 1 1 2 2 1 2{0,1,2,3}0max min 3 4 ( 3 ) (5 2 ) 0
jxux x u x x u x x
Optimal value = 10
Surrogate Dual
• Example:
• The surrogate dual is
• Solution:
1 2
1 2
1 2
1 2
min 3 4
3 0
5 2 0
, {0,1,2,3}
x x
x x
x x
x x
1 2 1 1 2 2 1 2{0,1,2,3}0max min 3 4 ( 3 ) (5 2 ) 0
jxux x u x x u x x
1 2
1 2
( , ) (2,5)
8x 11
, which yields
the surrogate
and bound (no duality gap in
25
this ca10 se)
u u
x
Optimal value = 10
Surrogate Dual
• The surrogate dual is also an inference dual.• It uses the same proof system as the linear programming
dual.• Deduce f(x) v from g(x) 0 when some surrogate
ug(x) 0 dominates f(x) v.– i.e., when ug(x) 0 implies f(x) v.– An incomplete proof system.
Surrogate Dual
• This yields the inference dual
– which is equivalent to the relaxation dual
0
max ( ) 0 ( ) for u
v ug x f x v x D
0
max min ( ) ( ) 0x Du
f x ug x
Aside: Subadditive Dual
• Suppose multiplication by nonnegative vector u is replaced by a general subadditive function u( ).– u(a + b) u(a) + u(b)– u is also nondecreasing and homogeneous.
• The surrogate dual becomes the subadditive dual.– Repeated linear combinations and rounding give rise
to a family of subadditive functions for integer programming.
• They provide a complete inference method.• There is no duality gap.
Lagrangean Dual
• The Lagrangean dual is another relaxation dual for inequality-constrained problems
min ( ) ( ) 0x D
f x g x
Lagrangean Dual
• The Lagrangean dual is another relaxation dual for inequality-constrained problems
• This time, remove the constraints entirely but “penalize” constraint violations in the objective function:
min ( ) ( ) 0x D
f x g x
0max min ( ) ( )
u x Df x ug x
Lagrangean Dual
• The Lagrangean dual is another relaxation dual for inequality-constrained problems
• This time, remove the constraints entirely but “penalize” constraint violations in the objective function:
– This is a relaxation since
min ( ) ( ) 0x D
f x g x
0max min ( ) ( )
u x Df x ug x
( ) ( ) ( ) for all feasible f x ug x f x x
0 0
Lagrangean Dual
• The Lagrangean dual provides a weaker bound than than the surrogate dual.– But it can be solved by steepest ascent search.
• Write the dual
as
– Then (u) is concave, and its subgradient is g(x).
0max min ( ) ( )
u x Df x ug x
0
where max ( ) ( ) min ( ) ( )x Du
u u f x ug x
Lagrangean Dual
• Example:
• The dual is
where
1 2
1 2
1 2
1 2
min 3 4
3 0
5 2 0
, {0,1,2,3}
x x
x x
x x
x x
1 2
1 2, 0
max ( , )u u
u u
1 2 1 2 1 1 2 2 1 2{0,1,2,3}( , ) min 3 4 ( 3 ) (5 2 )
jxu u x x u x x u x x
Lagrangean Dual
• Example:
• The dual is
where
• Solution:
1 2
1 2
1 2
1 2
min 3 4
3 0
5 2 0
, {0,1,2,3}
x x
x x
x x
x x
1 2
1 2, 0
max ( , )u u
u u
1 2 1 2 1 1 2 2 1 2{0,1,2,3}( , ) min 3 4 ( 3 ) (5 2 )
jxu u x x u x x u x x
5 13 21 2 7 7 7
27
with optimal value
Duality gap
( , ) ( , ) 9
10 9of
u u
Lagrangean Dual
• The Lagrangean and surrogate duals appear unrelated.– But when viewed as inference duals, they are closely
related.• The Lagrangean dual uses a slightly weaker proof
system.– It follows that it yields a weaker bound.
Lagrangean Dual
• Deduce f(x) v from g(x) 0 when some surrogate ug(x) 0 dominates f(x) v.– i.e., some surrogate ug(x) 0 dominates v f(x) 0.
Lagrangean Dual
• Deduce f(x) v from g(x) 0 when some surrogate ug(x) 0 dominates f(x) v.– i.e., some surrogate ug(x) 0 dominates v f(x) 0.
• In the surrogate dual, “dominates” meansug(x) 0 implies v f(x) 0.
Lagrangean Dual
• Deduce f(x) v from g(x) 0 when some surrogate ug(x) 0 dominates f(x) v.– i.e., some surrogate ug(x) 0 dominates v f(x) 0.
• In the surrogate dual, “dominates” meansug(x) 0 implies v f(x) 0.
• In the Lagrangean dual, it means something stronger:ug(x) v f(x).
– i.e., v f(x) + ug(x)
Lagrangean Dual
• Deduce f(x) v from g(x) 0 when some surrogate ug(x) 0 dominates f(x) v.– i.e., some surrogate ug(x) 0 dominates v f(x) 0.
• In the surrogate dual, “dominates” meansug(x) 0 implies v f(x) 0.
• In the Lagrangean dual, it means something stronger:ug(x) v f(x).
– i.e., v f(x) + ug(x)• So the resulting inference dual is the Lagrangean dual:
0
0
max ( ) ( ) for all
max min ( ) ( )
x Du
uv v f x ug x
x ug x
x D
f
Nogood-Based Search
• The inference dual provides the basis for nogood-based search.– Nogoods rule out solutions already examined.
• …and possibly other solutions that are no better.
Nogood-Based Search
• The inference dual provides the basis for nogood-based search.– Nogoods rule out solutions already examined.
• …and possibly other solutions that are no better.– Search proceeds by enumerating partial assignments
to the variables.– The next partial assignment must satisfy nogoods
generated so far.
Nogood-Based Search
• The inference dual provides the basis for nogood-based search.– Nogoods rule out solutions already examined.
• …and possibly other solutions that are no better.– Search proceeds by enumerating partial assignments
to the variables.– The next partial assignment must satisfy nogoods
generated so far.• The nogoods are obtained by analyzing the inference
dual of the restricted problem that results from each partial assignment.
Nogood-Based Search
• Solve:
• In each iteration, formulate a restriction of the problem:
min ( )x D
f x C
min ( )x D
f x
C B
Partial assignment that fixes some of the variables xj
Nogood-Based Search
• Solve:
• In each iteration, formulate a restriction of the problem:
– B may contain:• Branching constraints (in branching methods)• Solution of master problem (in Benders
decomposition)
min ( )x D
f x C
min ( )x D
f x
C B
Nogood-Based Search
• Let solve the inference dual of the restriction:
– Let fix the variables in B that serve as premises in
max ( ) ( )p P
f x f x v
C BP
( , )v P
BP
Nogood-Based Search
• Let solve the inference dual of the restriction:
– Let fix the variables in B that are needed as premises in
– Create the nogood
max ( ) ( )p P
f x f x v
C BP
( , )v P
BP
( )f x v B
Nogood-Based Search
• Let solve the inference dual of the restriction:
– Let fix the variables in B that serve as premises in
– Create the nogood
• The next partial assignment B must be consistent with all nogoods generated so far.– This avoids partial assignments already examined.
• …and others that are no better.
max ( ) ( )p P
f x f x v
C BP
( , )v P
BP
( )f x v B
Linear Programming: Nogood-Based Search
• Solve 0
min ( ) ( )F F
U
F U F Ux D
x
f x cx g x Ax b
Problem becomes linear subproblem when xF is fixed to .Fx
Linear Programming: Nogood-Based Search
• Solve
• Nogood-based search becomes Benders decomposition.– The nogoods are formed by solving the linear
programming dual of the problem that results when xF is fixed.
0
min ( ) ( )F F
U
F U F Ux D
x
f x cx g x Ax b
Problem becomes linear subproblem when xF is fixed to .Fx
Linear Programming: Nogood-Based Search
• Solve
• Let solve LP dual of the subproblem:
0
min ( ) ( )F F
U
F U F Ux D
x
f x cx g x Ax b
u
0
max ( )Fu
u b g x uA c
Linear Programming: Nogood-Based Search
• Solve
• Let solve LP dual of the subproblem:
– Then by strong duality ( = optimal value):
0
min ( ) ( )F F
U
F U F Ux D
x
f x cx g x Ax b
u
( ) ( )F Fz f x u b g x
0
max ( )Fu
u b g x uA c
z
Linear Programming: Nogood-Based Search
• Solve
• Let solve LP dual of the subproblem:
– Then by strong duality ( = optimal value):
– And since is dual feasible for any xF, weak duality implies the Benders cut
0
min ( ) ( )F F
U
F U F Ux D
x
f x cx g x Ax b
u
Fu ( ) ( )F Fz f x u b g x
( ) ( )F Fz f x u b g x
0
max ( )Fu
u b g x uA c
z
SAT: Nogood-Based Search
• Consider the propositional satisfiability problem (SAT), which can be written
• Branching search (DPL) with clause learning can be viewed as a nogood-based search.– Unit resolution (unit clause rule) is applied at every
node of the search tree.– The nogoods are conflict clauses…
{T,F}
min 0jx
C Set of logical clauses
SAT: Nogood-Based Search
1 5
SAT problem
+ partial assignment
( , , ) (F,F,F,F,F)
is infeasible ( )
x x
v
Solution of inference dual is a unit resolution proof of infeasibility.
SAT: Nogood-Based Search
1 5
SAT problem
+ partial assignment
( , , ) (F,F,F,F,F)
is infeasible
x x
Solution of inference dual is a unit resolution proof of infeasiblity.
Proof remains valid when only x1 and x5 are fixed to F.
SAT: Nogood-Based Search
1 5
SAT problem
+ partial assignment
( , , ) (F,F,F,F,F)
is infeasible
x x
Solution of inference dual is a unit resolution proof of infeasiblity.
Proof remains valid when only x1 and x5 are fixed to F.
So we have the nogood
1 5( , ) (F,F) ( )x x f x
SAT: Nogood-Based Search
1 5
SAT problem
+ partial assignment
( , , ) (F,F,F,F,F)
is infeasible
x x
Solution of inference dual is a unit resolution proof of infeasiblity.
Proof remains valid when only x1 and x5 are fixed to F.
So we have the nogood
or conflict clause 1 5x x
1 5( , ) (F,F) ( )x x f x
SAT: Nogood-Based Search
1 2
1 2
Next partial assignment
( , ) (F,T)
must satisfy previous
2 nogoods and
their resolvent
x x
x x
SAT: Nogood-Based Search
At this point the
accumulated nogoods
are unsatisfiable
and the search
is exhaustive
Domain Filtering Duals
• Since a domain filter is an inference method, it defines an inference dual and a nogood-based search method.
– Let [Lj,Uj] be the domain of xj.
– So x [L,U] where L = (L1,…,Ln), U = (U1,…,Un)
– Then the inference dual can be defined
where P contains the domain filtering operations.
max ( ( ) ) max ( ) ( [ , ])p P P P
v f x v f x x L U
C CP P
Domain Filtering Duals
• Suppose we are branching.– Let B contain the current branching constraints.– Let solve the inference dual.– Let contain the constraints in B used as premises
in the filtering operation– We have the nogood
( , )v P
BP
( )f x v B
Domain Filtering Duals
• Example: assign jobs to machines and schedule them to minimize makespan.
{1, , }, all
[ , ], all
, all min
disjunctive ( ),( ) , all
j
j
j j j
j x j
x m jj j ij j
s r d j
v s p jv
s x i p x i i
Start time of job jprocessing time of job j on machine i
Machine assigned to job j
Time window for job j
Planning & Scheduling: Filtering Dual
Suppose there are 5 jobs and 2 machines:
We will search overmachine assignments x.
Planning & Scheduling: Filtering Dual
Suppose there are 5 jobs and 2 machines:
We will search overmachine assignments x.
At the current point in the search, we have assigned jobs 1,2,3,5 to machine A and job 4 to machine B.
Planning & Scheduling: Filtering Dual
Suppose there are 5 jobs and 2 machines:
We will search overmachine assignments x.
At the current point in the search, we have assigned jobs 1,2,3,5 to machine A and job 4 to machine B.
Min makespan on machine B is 5.
Planning & Scheduling: Filtering Dual
Suppose there are 5 jobs and 2 machines:
We will search overmachine assignments x.
At the current point in the search, we have assigned jobs 1,2,3,5 to machine A and job 4 to machine B.
Min makespan on machine B is 5.
Let’s compute min makespan on machine A…
Planning & Scheduling: Filtering Dual
Job 1
Job 2
Job 3
Job 5
Time window
Schedule
10
We find a feasible schedule with makespan 10.
Machine A
Planning & Scheduling: Filtering Dual
Job 1
Job 2
Job 3
Job 5
Time window
Schedule
9
We found a feasible schedule with makespan 10.
To prove optimality, we check whether a makespan of 9 is possible.
Machine A
Planning & Scheduling: Filtering Dual
Job 1
Job 2
Job 3
Job 5
Time window
Schedule
9
Edge finding discovers that job 2 must precede jobs 3 and 5.
Machine A
Planning & Scheduling: Filtering Dual
Job 1
Job 2
Job 3
Job 5
Time window
Schedule
9
Edge finding discovers that job 2 must precede jobs 3 and 5.
Since this reduces domain of s2 to empty set, there is no feasible solution with makespan 9.
Machine A
Planning & Scheduling: Filtering Dual
Job 1
Job 2
Job 3
Job 5
Time window
Schedule
9This edge finding procedure solves the inference dual of the scheduling problem.
Machine A
P
Planning & Scheduling: Filtering Dual
Job 1
Job 2
Job 3
Job 5
Time window
Schedule
9This edge finding procedure solves the inference dual of the scheduling problem.
derives infeasibility from the assignments Bof jobs 1,2,3,5 to machine A.
Machine A
P
P
Planning & Scheduling: Filtering Dual
Job 1
Job 2
Job 3
Job 5
Time window
Schedule
9This edge finding procedure solves the inference dual of the scheduling problem.
derives infeasibility from the assignments Bof jobs 1,2,3,5 to machine A.
Is there a smaller set of assignments that is sufficient to prove infeasibility?
Machine A
P
P
B
Planning & Scheduling: Filtering Dual
Job 1
Job 2
Job 3
Job 5
Time window
Schedule
9
Only jobs 2, 3, 5 were involved in this deduction. So we have the nogood:
Machine A
2 3 5 10x x x A v
B
Planning & Scheduling: Filtering Dual
Job 1
Job 2
Job 3
Job 5
Time window
Schedule
9
Only jobs 2, 3, 5 were involved in this deduction. So we have the nogood:
To improve the solution, the search must avoid this assignment.
Machine A
2 3 5 10x x x A v
Planning & Scheduling: Filtering Dual
Job 1
Job 2
Job 3
Job 5
Time window
Schedule
9
Only jobs 2, 3, 5 were involved in this deduction. So we have the nogood:
Since the nogoods contain a fixed set of variables x, this is a generalization of Benders decomposition
Machine A
2 3 5 10x x x A v
Constraint Dual
• The constraint dual is neither a relaxation dual nor an inference dual.
• However, one can define a relaxation dual for a constraint dual formulation.– Mini-bucket elimination is a special case.
• This is best explained by example…
Constraint Dual
• Consider the 0-1 problem
1 2 3 4
1 2 3
1 2 4
2 3 4
min 2 3 4
1
1
1
{0,1}j
x x x x
x x x
x x x
x x x
x
Constraint Dual
• Consider the 0-1 problem
– Solution:
1 2 3 4
1 2 3
1 2 4
2 3 4
min 2 3 4
1
1
1
{0
2
,1}j
x x x x
x x x
x x x
x x x
x
1 4( , , ) (0,1,0,0)x x
Constraint Dual
• One form of constraint dual (ignoring the objective function)is
1 2 3 4
1 2 3
1 2 4
2 3 4
min 2 3 4
1
1
1
{0,1}j
x x x x
x x x
x x x
x x x
x
1 21 11 2 32 2 21 33 31 34 4
1 1 11 2 32 2 21 2 43 3 32 3 4
( , , )
( , , )
( , , )
x x
x x x
x x
x x
x x x D
x x x D
x x x D
where 3{0,1} {(0,0,0)}D
Standardize apart variables in different constraints and equate them in binary constraints.
Constraint Dual
12x 1
3x
21x
11x
22x
32x
24x
34x3
3x
Dependency graph for constraint dual:
1 2 3 4
1 2 3
1 2 4
2 3 4
min 2 3 4
1
1
1
{0,1}j
x x x x
x x x
x x x
x x x
x
Induced width of dependency graph is 3.
This indicates complexity of solving the problem by nonserial dynamic programming.
Constraint Dual
1 2 3 4
1 2 3
1 2 4
2 3 4
min 2 3 4
1
1
1
{0,1}j
x x x x
x x x
x x x
x x x
x
3x
2x1x
4x
Dependency graph of original problem also has induced width 3:
Constraint Dual
• One way to relax the dynamic programming model is to remove edges from the dependency graph.– and form “mini-buckets”
1 2 3 4
1 2 3
1 2 4
2 3 4
min 2 3 4
1
1
1
{0,1}j
x x x x
x x x
x x x
x x x
x
1 1 2 3 2 1 2 4 3 2 3 4
1 2 3 1 2 31 1 2 3
1 2 4 1 2 42 1 2 4
2 3 4 2 3 43 2 3 4
min ( , , ) ( , , ) ( , , )
2 3 if 1( , , )
otherwise
0 0 4 if 1( , , )
otherwise
0 0 0 if 1( , , )
otherwise
f x x x f x x x f x x x
x x x x x xf x x x
x x x x x xf x x x
x x x x x xf x x x
Constraint Dual
1 1 2 3 2 1 2 4 3 2 3 4
1 2 3 1 2 31 1 2 3
1 2 4 1 2 42 1 2 4
2 3 4 2 3 43 2 3 4
min ( , , ) ( , , ) ( , , )
2 3 if 1( , , )
otherwise
0 0 4 if 1( , , )
otherwise
0 0 0 if 1( , , )
otherwise
f x x x f x x x f x x x
x x x x x xf x x x
x x x x x xf x x x
x x x x x xf x x x
• The problem separates into 3 problems with induced width 2:
3x
2x1x 2x
1x
4x 3x
2x
4x
Constraint Dual
1 1 2 3 2 1 2 4 3 2 3 4
1 2 3 1 2 31 1 2 3
1 2 4 1 2 42 1 2 4
2 3 4 2 3 43 2 3 4
min ( , , ) ( , , ) ( , , )
2 3 if 1( , , )
otherwise
0 0 4 if 1( , , )
otherwise
0 0 0 if 1( , , )
1 0
o
0 1
t
f x x x f x x x f x x x
x x x x x xf x x x
x x x x x xf x x x
x x x x x xf x x x
herwise
• But the resulting bound is weak
3x
2x1x 2x
1x
4x 3x
2x
4x
Constraint Dual
Another approach is to delete edges from the dependency graph of the constraint dual.
1 2 3 4
1 2 3
1 2 4
2 3 4
min 2 3 4
1
1
1
{0,1}j
x x x x
x x x
x x x
x x x
x
12x 1
3x
21x
11x
22x
32x
24x
34x3
3x
Constraint Dual
12x 1
3x
21x
11x
22x
32x
24x
34x3
3x
Another approach is to delete edges from the dependency graph of the constraint dual.
1 2 3 4
1 2 3
1 2 4
2 3 4
min 2 3 4
1
1
1
{0,1}j
x x x x
x x x
x x x
x x x
x
The previous mini-bucket scheme deletes all edges connecting variables.
Constraint Dual
Another approach is to delete edges from the dependency graph of the constraint dual.
1 2 3 4
1 2 3
1 2 4
2 3 4
min 2 3 4
1
1
1
{0,1}j
x x x x
x x x
x x x
x x x
x
By deleting only one edge…
12x 1
3x
21x
11x
22x
32x
24x
34x3
3x
Constraint Dual
Another approach is to delete edges from the dependency graph of the constraint dual.
1 2 3 4
1 2
1 2 4
2
3
3 4
min 2 3 4
1
1
1
{0,1}j
x x x x
x x
x x
y
x
x x x
x
By deleting only one edge…one can reduce the induced width to 2.
12x 1
3x
21x
11x
22x
32x
24x
34x3
3x
Constraint Dual
Another approach is to delete edges from the dependency graph of the constraint dual.
1 2 3 4
1 2
1 2 4
2 3 4
3
min 2 3 4
1
1
1
{0,1
2
}j
x x x x
x x
x x x
x x
x
y
x
By deleting only one edge…one can reduce the induced width to 2.
The bound is now tight.
12x 1
3x
21x
11x
22x
32x
24x
34x3
3x
Constraint Dual
• One can now define a relaxation dual: choose a set of eliminated edges that provides the tightest bound.
• The choice of edges parameterizes the relaxation.
The dual can be solved by various heuristics, such as local search.
12x 1
3x
21x
11x
22x
32x
24x
34x3
3x
Primal-Dual Methods
• Primal-dual inference methods– Branch-and-cut for integer programming
• Combine branching (search over restrictions) with inferred cutting planes.
– Standard CP methods• Combine branching with filtering.
Primal-Dual Methods
• Primal-Dual Relaxation Methods– Branch-and-bound with Lagrangean relaxation.
• Solve Lagrangean relaxation at each node of search tree.
– Interleave search over restrictions with search over relaxations.
• Dual simplex method for linear programming.• “Out-of-kilter” method for network flows.• Dual ascent method for integer programming.
Duality of Duals?
• Why can so many duals be interpreted as both relaxation duals and inference duals?– Particularly math programming duals.
• Is there a duality of duals?
Duality of Duals?
• Why can so many duals be interpreted as both relaxation duals and inference duals?– Particularly math programming duals.
• Is there a duality of duals?
• No.– But there is a formal relationship between the duals…
Duality of Duals?
• A relaxation dual in which there is a finite algorithm for solving the relaxation is an inference dual.– The solution algorithm can be viewed as inferring the
best bound.• An inference dual in which the proofs are parameterized
is a relaxation dual.– The proof of the best bound can be viewed as solving
the tightest relaxation.
Duality of Duals?
• Math programming duals tend to be conceived as relaxation duals.– There is always an algorithm for solving the
relaxation, otherwise the dual would be useless.– So math programming duals tend to be both
relaxation and inference duals.• Inference methods are not ordinarily parameterized.
– So inference duals are not necessarily relaxation duals.
Top Related