linear programming - ETH Z · theorem: every feasible, bounded linear program in equational form...

linear programming

David Steurer

Algorithms, Computing, and Probability, Fall 2018, ETH Zürich

linear programminggiven: linear function (objective) and system of linearinequalities (constraints)

find: optimal solution that maximizes objective among allsolutions that satisfy the constraints ( feasible solutions)

c x⊺

{Ax ⩽ b}

x∗

s t

3/3

0/2 1/4

2/2

3/3

2/2

2/3

3/3

o

p

q

r

application: network flow

network flowgiven: dir. graph with edge cap. ,source and target

find: maximum flow from to in within capacities

G = (V , E) c: E → R⩾0

s ∈ V t ∈ V

s t G c

s t

3/3

0/2 1/4

2/2

3/3

2/2

2/3

3/3

o

p

q

r




linear programming formulationvariables: for all

G = (V , E) c: E → R⩾0

s ∈ V t ∈ V

s t G c

f ⩾ 0e e ∈ E

s t

3/3

0/2 1/4

2/2

3/3

2/2

2/3

3/3

o

p

q

r





capacity: for all

G = (V , E) c: E → R⩾0

s ∈ V t ∈ V

s t G c

f ⩾ 0e e ∈ E

f ⩽ c(e)e e ∈ E

s t

3/3

0/2 1/4

2/2

3/3

2/2

2/3

3/3

o

p

q

r





capacity: for all

conservation: for

G = (V , E) c: E → R⩾0

s ∈ V t ∈ V

s t G c

f ⩾ 0e e ∈ E

f ⩽ c(e)e e ∈ E

f = f∑e∈δ (u)G+ e ∑e∈δ (u)

G− e u ∈ V ∖ {s, t}

s t

3/3

0/2 1/4

2/2

3/3

2/2

2/3

3/3

o

p

q

r





capacity: for all

conservation: for

objective: (net flow out of source)

G = (V , E) c: E → R⩾0

s ∈ V t ∈ V

s t G c

f ⩾ 0e e ∈ E

f ⩽ c(e)e e ∈ E

f = f∑e∈δ (u)G+ e ∑e∈δ (u)

G− e u ∈ V ∖ {s, t}

f − f∑e∈δ (s)G+ e ∑e∈δ (s)

G− e

geometry of linear programs


hyperplane: solutions to linear equation {x ∣ a x = b}⊺


hyperplane: solutions to linear equation

half-space: solutions to linear inequality

{x ∣ a x = b}⊺

{x ∣ a x ⩽ b}⊺




polyhedron: finite intersection of half-spaces (feasible region of linear program)

lemma: every polyhedron is convex (contains the linesegment between any two of its points)

{x ∣ a x = b}⊺

{x ∣ a x ⩽ b}⊺

{x ∣ Ax ⩽ b}






proof:

{x ∣ a x = b}⊺

{x ∣ a x ⩽ b}⊺

{x ∣ Ax ⩽ b}






proof:

corollary: set of optima of linear program is convex

{x ∣ a x = b}⊺

{x ∣ a x ⩽ b}⊺

{x ∣ Ax ⩽ b}

basic feasible solution

consider linear program with constraints {Ax ⩽ b}


consider linear program with constraints

definition: feasible solution is basic if it satisfies lin.-indep. constraints from with equality

{Ax ⩽ b}

x ∈ Rn n

{Ax ⩽ b}




In other words, is basic feasible solution (BFS) if and subset of rows such that and has rank

{Ax ⩽ b}

x ∈ Rn n

{Ax ⩽ b}

x Ax ⩽ b

∃ S n A x = bS S AS n





claim: linear program with constraints has BFSs

{Ax ⩽ b}

x ∈ Rn n

{Ax ⩽ b}

x Ax ⩽ b


m ⩽ (nm)






proof: every subset gives rise to at most one BFS (why?)

{Ax ⩽ b}

x ∈ Rn n

{Ax ⩽ b}

x Ax ⩽ b


m ⩽ (nm)







geometry: BFS is corner of polyhedron

{Ax ⩽ b}

x ∈ Rn n

{Ax ⩽ b}

x Ax ⩽ b


m ⩽ (nm)

P = {x ∣ Ax ⩽ b}







geometry: BFS is corner of polyhedron

for every BFS , linear objective such that is unique opt.

{Ax ⩽ b}

x ∈ Rn n

{Ax ⩽ b}

x Ax ⩽ b


m ⩽ (nm)

P = {x ∣ Ax ⩽ b}

x ∃ x


claim: no BFS is strict convex combination of two pts in P


claim: no BFS is strict convex combination of two pts in

proof:

P

basic feasible solutions are optimal

theorem: every feasible, bounded linear program inequational form has a BFS as optimum



there may be (infinitely) many optima,but at least one of them is a BFS!




recall: equational form without loss of generality

finite (but exponential) algorithm for linear programming:enumerate all basic feasible solutions (how?) and output onewith best objective value




recall: equational form without loss of generality

finite (but exponential) algorithm for linear programming:enumerate all basic feasible solutions (how?) and output onewith best objective value

proof of theorem: suppose objective is to maximize subject to constraints ; assume LP bounded

c x⊺

{Ax = b, x ⩾ 0}



enough to show: for every feasible , can find BFS withobjective

c x⊺

{Ax = b, x ⩾ 0}

x ∈ R0n

⩾ c x⊺0




among all feasible with , choose one withmaximum number of zero coordinates (will show: is BFS)

c x⊺

{Ax = b, x ⩾ 0}

x ∈ R0n

⩾ c x⊺0

x c x ⩾ c x⊺ ⊺0 x~

x~





w.l.o.g.: exactly first coordinates of are non-zero

c x⊺

{Ax = b, x ⩾ 0}

x ∈ R0n

⩾ c x⊺0

x c x ⩾ c x⊺ ⊺0 x~

x~

k ⩾ 0 x~





w.l.o.g.: exactly first coordinates of are non-zero

claim: first columns of are linearly-independent

c x⊺

{Ax = b, x ⩾ 0}

x ∈ R0n

⩾ c x⊺0

x c x ⩾ c x⊺ ⊺0 x~

x~

k ⩾ 0 x~

k A

theorem: every feasible, bounded linear program in equational form has a BFS as optimum


if claim holds, then is indeed BFS (why?)

k A

x~



if claim holds, then is indeed BFS (why?)

proof of claim: (by contradiction) otherwise, with and

k A

x~

∃w ≠ 0Aw = 0 w , … , w = 0k+1 n




case 1: if and , then solutions for are feasible and have unbounded objective;

k A

∃w ≠ 0Aw = 0 w , … , w = 0k+1 n

c w > 0⊺ w ⩾ 0 x = + t ⋅ w(t) x~

t ⩾ 0




case 1: if and , then solutions for are feasible and have unbounded objective;contradicting boundedness of LP

k A

∃w ≠ 0Aw = 0 w , … , w = 0k+1 n

c w > 0⊺ w ⩾ 0 x = + t ⋅ w(t) x~

t ⩾ 0




case 2: if and has neg. coord., then suchthat is feasible with and more zerocoords. than ;

k A

∃w ≠ 0Aw = 0 w , … , w = 0k+1 n

c w ⩾ 0⊺ w ∃t ⩾ 0x = + t ⋅ w′ x~ c x ⩾ c x⊺ ′ ⊺

0

x~




case 2: if and has neg. coord., then suchthat is feasible with and more zerocoords. than ; contradicting choice of

k A

∃w ≠ 0Aw = 0 w , … , w = 0k+1 n

c w ⩾ 0⊺ w ∃t ⩾ 0x = + t ⋅ w′ x~ c x ⩾ c x⊺ ′ ⊺

0

x~ x~

algorithms for linear programming


simplex method [Dantzig’46]

move along adjacent corners of the feasible polyhedron until anoptimal solution is found




works well in practice but has super-polynomial time in theworst case





ellipsoid method [Shor–Judin–Nemirovski’70, Khachyian’79]

first polynomial-time algorithm (but not practical)





ellipsoid method [Shor–Judin–Nemirovski’70, Khachyian’79]

first polynomial-time algorithm (but not practical)

interior-point methods [Karmarkar’84, …]

best-known worst-case running time; also practical

lots of recent works [e.g., Lee–Sidford’14]; even leading to fasternetwork flow algorithms [e.g., Madry]

linear programming (LP) as algorithmic techniqueone of the most powerful and versatile tools for algorithm design


for many problems, the algorithms with the best known provableguarantees rely on LP as a subroutine

examples

traveling salesman problemnetwork design (e.g., Steiner tree)edge-disjoint pathscompressed sensingnonnegative matrix factorization

…



typical pattern for such algorithms

construct linear program (relaxation)�. find an optimal solution to the linear program�. post-process to get sol. for orig. problem (rounding)�.

x∗

x∗



typical pattern for such algorithms

construct linear program (relaxation)�. find an optimal solution to the linear program�. post-process to get sol. for orig. problem (rounding)�.

typically, most of the “analysis work” is in step 3 and most ofthe “algorithmic work” is in step 2

x∗

x∗

computational complexity of linear programmingLP as decision problem: given a system of linear inequalities,decide if it is satisfiable or not


LP in NP: if a system of linear inequalities is satisfiable, thereis a polynomial-size certificate for that



LP in coNP: if a system of linear inequalities is not satisfiable,there is a polynomial-size certificate for that



LP in coNP: if a system of linear inequalities is not satisfiable,there is a polynomial-size certificate for that

LP in P: there is a polynomial-time algorithm to decide thesatisfiability of systems of linear inequalities

bounds on LP solutions(or LP in NP)

notation: (binary) encoding size of object

theorem: for every bounded linear program , there exists

an optimal solution such that

⟨X⟩ = X

L

x∗ ⟨x ⟩ ⩽ ⟨L⟩∗ O(1)




proof: it is enough to bound the encoding size of basic feasiblesolutions because there always exists a basic feasible solutionthat is optimal

⟨X⟩ = X

L

x∗ ⟨x ⟩ ⩽ ⟨L⟩∗ O(1)




proof: it is enough to bound the encoding size of basic feasiblesolutions because there always exists a basic feasible solutionthat is optimal

in the next slides, we will bound the encoding size of basicfeasible solutions

⟨X⟩ = X

L

x∗ ⟨x ⟩ ⩽ ⟨L⟩∗ O(1)

lemma: every satisfiesA ∈ Zn×n

⟨detA⟩ ⩽ O(⟨A⟩)

lemma: every satisfies

proof:

A ∈ Zn×n


∣detA∣ = ∣ sign(π) a ∣∑π∈Sn∏i=1

ni,π(i)


proof:

A ∈ Zn×n



ni,π(i)

⩽ n! ⋅ 2⟨A⟩ = 2⟨A⟩+O(n log n)


proof: since ,

A ∈ Zn×n


⟨A⟩ ⩾ n2


ni,π(i)

⩽ n! ⋅ 2⟨A⟩ = 2⟨A⟩+O(n log n) ⩽ 2O(⟨A⟩) □

lemma: every basic feasible solution of satisfiesx~ L∀j ∈ [n]. ⟨ ⟩ ⩽ O(⟨A⟩ + ⟨b⟩ + n )x~j

2

let be an LP in equational form

suppose has linear constraints with and

lemma: every basic feasible solution of satisfies

L

L {Ax = b,x ⩾ 0}b ∈ Zm A ∈ Zm×n

x~ L∀j ∈ [n]. ⟨ ⟩ ⩽ O(⟨A⟩ + ⟨b⟩ + n )x~j

2




proof: since is a BFS, it is the solution of a non-singularlinear system , where each equations is either oneof the equations in or of the form

L

L {Ax = b,x ⩾ 0}b ∈ Zm A ∈ Zm×n

x~ L∀j ∈ [n]. ⟨ ⟩ ⩽ O(⟨A⟩ + ⟨b⟩ + n )x~j

2

x~

{ x = }A~

b~

{Ax = b} x = 0i





by Cramer’s rule, , where is obtainedfrom by replacing the -th column with

L

L {Ax = b,x ⩾ 0}b ∈ Zm A ∈ Zm×n

x~ L∀j ∈ [n]. ⟨ ⟩ ⩽ O(⟨A⟩ + ⟨b⟩ + n )x~j

2

x~

{ x = }A~

b~

{Ax = b} x = 0i

= det / detx~j A~

j A~

A~

j

A~

j b~





by Cramer’s rule, , where is obtainedfrom by replacing the -th column with

therefore,

L

L {Ax = b,x ⩾ 0}b ∈ Zm A ∈ Zm×n

x~ L∀j ∈ [n]. ⟨ ⟩ ⩽ O(⟨A⟩ + ⟨b⟩ + n )x~j

2

x~

{ x = }A~

b~

{Ax = b} x = 0i

= det / detx~j A~

j A~

A~

j

A~

j b~

⟨ ⟩ ⩽ ⟨det ⟩ + ⟨det ⟩ ⩽ O(⟨A⟩ + ⟨b⟩ + n )x~j A~

j A~ 2 □

duality and Farkas lemma(or LP in coNP)

certificates of satisfiability and unsatisfiabilitywe saw: the satisfiability of a system of linear inequalitiesalways has a polynomial-size certificate

certificates of satisfiability and unsatisfiabilitywe saw: the satisfiability of a system of linear inequalitiesalways has a polynomial-size certificate

how could we certify that a system of linear inequalities isunsatisfiable?

unsatisfiability of systems of linear equations

theorem: A linear system is unsatisfiable iff thefollowing linear system is satisfiable

{Ax = b}

{A y = 0, b y = 1}⊺ ⊺



idea: for every , we can derive the implied equation from the equations

{Ax = b}

{A y = 0, b y = 1}⊺ ⊺

y

{y Ax = y b}⊺ ⊺ {Ax = b}



idea: for every , we can derive the implied equation from the equations

the theorem says that is unsatisfiable if and only ifwe can derive in this way the syntactically unsatisfiableequation

{Ax = b}

{A y = 0, b y = 1}⊺ ⊺

y

{y Ax = y b}⊺ ⊺ {Ax = b}

{Ax = b}

{0 x = 1}⊺

unsatisfiability of systems of linear inequalities

Farkas lemma (form I): is unsatisfiable iff thefollowing system of linear inequalities is satisfiable

{Ax ⩽ b}

{A y = 0, b y = −1, y ⩾ 0}⊺ ⊺



idea: for every , we can derive the implied constraint from the constraints ;

{Ax ⩽ b}

{A y = 0, b y = −1, y ⩾ 0}⊺ ⊺

y ⩾ 0{y Ax ⩽ y b}⊺ ⊺ {Ax ⩽ b}



idea: for every , we can derive the implied constraint from the constraints ; if satisfies

above conditions, this implied constraint itself is unsatisfiable

{Ax ⩽ b}

{A y = 0, b y = −1, y ⩾ 0}⊺ ⊺

y ⩾ 0{y Ax ⩽ y b}⊺ ⊺ {Ax ⩽ b} y





equivalent form II: a system is unsatisfiableif and only if the system is satisfiable

{Ax ⩽ b}

{A y = 0, b y = −1, y ⩾ 0}⊺ ⊺

y ⩾ 0{y Ax ⩽ y b}⊺ ⊺ {Ax ⩽ b} y

{Ax = b,x ⩾ 0}{A y ⩾ 0, b y = −1}⊺ ⊺





equivalent form II: a system is unsatisfiableif and only if the system is satisfiable

complexity interpretation: LP in coNP

{Ax ⩽ b}

{A y = 0, b y = −1, y ⩾ 0}⊺ ⊺

y ⩾ 0{y Ax ⩽ y b}⊺ ⊺ {Ax ⩽ b} y

{Ax = b,x ⩾ 0}{A y ⩾ 0, b y = −1}⊺ ⊺

geometry interpretation

columns of matrix generate conea ,… , a1 n A

C = {x ⋅ a +⋯+ x ⋅ a ∣ x ,… ,x ⩾ 0}1 1 n n 1 n


columns of matrix generate cone

system is unsat. iff

a ,… , a1 n A

C = {x ⋅ a +⋯+ x ⋅ a ∣ x ,… ,x ⩾ 0}1 1 n n 1 n

{Ax = b,x ⩾ 0} b C∈




geometry obs.: iff halfspace with but

a ,… , a1 n A

C = {x ⋅ a +⋯+ x ⋅ a ∣ x ,… ,x ⩾ 0}1 1 n n 1 n

{Ax = b,x ⩾ 0} b C∈

b C∈ ∃ H = {x ∣ y x ⩾ 0}⊺

a ,… , a ∈ H1 n b H∈




geometry obs.: iff halfspace with but

in other words: iff with but

a ,… , a1 n A

C = {x ⋅ a +⋯+ x ⋅ a ∣ x ,… ,x ⩾ 0}1 1 n n 1 n

{Ax = b,x ⩾ 0} b C∈

b C∈ ∃ H = {x ∣ y x ⩾ 0}⊺

a ,… , a ∈ H1 n b H∈

b C∈ ∃y A y ⩾ 0⊺

b y < 0⊺

certificates for optimization problemswe saw: systems of linear inequalities have polynomial-sizecertificates for both satisfiability and unsatisfiability(namely, basic feasible solutions for the system or its dual)


for optimization problems, our goal is to certify lower andupper bounds on the optimal objective value



maximize c x subject to Ax ⩽ b and x ⩾ 0 (P)⊺



to certify lower bounds, can use basic feasible solutions for (P)





how to certify upper bounds?





how to certify upper bounds?

idea: to certify upper bounds for (P), we derive bycombining the constraints


{c x ⩽ γ}⊺

{Ax ⩽ b,x ⩾ 0}

certificates for optimization problemsfor optimization problems, our goal is to certify lower andupper bounds on the optimal objective value


concretely, if satisfies , then we can derive theinequality for


{c x ⩽ γ}⊺

{Ax ⩽ b,x ⩾ 0}

y ⩾ 0 A y ⩾ c⊺

{c x ⩽ γ}⊺ γ = b y⊺




the following optimization problem, searches for the best suchupper bound on (P)


{c x ⩽ γ}⊺

{Ax ⩽ b,x ⩾ 0}

y ⩾ 0 A y ⩾ c⊺

{c x ⩽ γ}⊺ γ = b y⊺

minimize b y subject to A y ⩾ c and y ⩾ 0 (D)⊺ ⊺




the following optimization problem, searches for the best suchupper bound on (P)

strong-duality theorem: (P) and (D) have same optimal value(assuming at least one is feasible)


{c x ⩽ γ}⊺

{Ax ⩽ b,x ⩾ 0}

y ⩾ 0 A y ⩾ c⊺

{c x ⩽ γ}⊺ γ = b y⊺


strong-duality theorem: following optimization problemshave same optimal value (assuming at least one is feasible)




by convention: infeasible maximization has optimal value and infeasible minimization has optimal value



−∞∞



special cases: max-flow min-cut, Hall’s theorem (bipartitematching), and Menger-type theorems (disjoint paths)



−∞∞



special cases: max-flow min-cut, Hall’s theorem (bipartitematching), and Menger-type theorems (disjoint paths)

weak-duality: optimal value of (P) optimal value of (D)(holds by construction)



−∞∞

⩽


observation: if one of (P) or (D) is unbounded, the other has tobe infeasible (by weak duality)





thus, can assume one of (P) or (D) is feasible and bounded





thus, can assume one of (P) or (D) is feasible and bounded

proof for the case that (P) is feasible and bounded:





for optimal value of (P) and , consider the system



γ ε ⩾ 0{Ax ⩽ b, c x ⩾ γ + ε, x ⩾ 0} (∗)⊺




for , system is infeasible



γ ε ⩾ 0{Ax ⩽ b, c x ⩾ γ + ε, x ⩾ 0} (∗)⊺

ε > 0 (∗)





by Farkas lemma,



γ ε ⩾ 0{Ax ⩽ b, c x ⩾ γ + ε, x ⩾ 0} (∗)⊺

ε > 0 (∗)

∃z ⩾ 0, y ⩾ 0A y ⩾ z ⋅ c and b y < z ⋅ (γ + ε)⊺ ⊺





by Farkas lemma,

since is feasible for , we must have



γ ε ⩾ 0{Ax ⩽ b, c x ⩾ γ + ε, x ⩾ 0} (∗)⊺

ε > 0 (∗)

∃z ⩾ 0, y ⩾ 0A y ⩾ z ⋅ c and b y < z ⋅ (γ + ε)⊺ ⊺

(∗) ε = 0 b y ⩾ z ⋅ γ⊺





by Farkas lemma,

since is feasible for , we must have

thus, ; by rescaling , we can assume ; now hasobjective value for (D)



γ ε ⩾ 0{Ax ⩽ b, c x ⩾ γ + ε, x ⩾ 0} (∗)⊺

ε > 0 (∗)

∃z ⩾ 0, y ⩾ 0A y ⩾ z ⋅ c and b y < z ⋅ (γ + ε)⊺ ⊺

(∗) ε = 0 b y ⩾ z ⋅ γ⊺

z > 0 y z = 1 y

< γ + ε □

Ellipsoid method(or LP in P)

Khachiyan’s achievement received anattention in the nonscientific press that is—toour knowledge—unpreceded in mathematics.Newspapers and journals like TheGuardian, Der Spiegel, NieuweRotterdamsche Courant, Nepszabadsag,The Daily Yomiuri wrote about the “majorbreakthrough in the solution of real-worldproblems”. The ellipsoid method evenjumped on the front page of The New YorkTimes: “A Soviet Discovery Rocks World ofMathematics” (November 7, 1979). Much ofthe excitement of the journalists was,however, due to exaggerations andmisinterpretations - see LAWLER (1980) foran account of the treatment of theimplications of the ellipsoid method in thepublic press.

(from textbook by Grötschel, Lovasz, and Schrijver)

relaxed feasibility problemgiven: , polyhedron

promise:

goal: find some point

R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n

∃y ∈ R . Ball(y, ε) ⊆ P ⊆ Ball(0,R)n

y ∈ P


promise:


will see later: polynomial-time algorithm for this problem canbe turned to polynomial-time algorithm for linear prog.

R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P


promise:


will see later: polynomial-time algorithm for this problem canbe turned to polynomial-time algorithm for linear prog.

bird’s eye view of ellipsoid methodcompute successively tighter outer-approximations for

R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

P

given: , polyhedron

promise:



concretely, sequence of ellipsoids such that

where and center of is in

R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

P

E ,… ,E ⊆ R0 Nn

E ,E ,⋯ ,E ⊇ P0 1 N

E = Ball(0,R)0 EN P

given: , polyhedron

promise:





key property: alg. ensures

R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

P

E ,… ,E ⊆ R0 Nn

E ,E ,⋯ ,E ⊇ P0 1 N

E = Ball(0,R)0 EN P

vol(E ) ⩽ e ⋅ vol(E )k+1−1/n

k

given: , polyhedron

promise:






consequence: number of iterations as

R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

P

E ,… ,E ⊆ R0 Nn

E ,E ,⋯ ,E ⊇ P0 1 N

E = Ball(0,R)0 EN P

vol(E ) ⩽ e ⋅ vol(E )k+1−1/n

k

N ⩽ poly(n, ⟨R⟩, ⟨ε⟩)

given: , polyhedron

promise:






consequence: number of iterations as

R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

P

E ,… ,E ⊆ R0 Nn

E ,E ,⋯ ,E ⊇ P0 1 N

E = Ball(0,R)0 EN P

vol(E ) ⩽ e ⋅ vol(E )k+1−1/n

k

N ⩽ poly(n, ⟨R⟩, ⟨ε⟩)

( ) = ⩽ ⩽ eRε n

vol(Ball(0,R))vol(Ball(0,ε))

vol(E )0vol(E )N −N/n

given: , polyhedron

promise:


finding the next ellipsoid

R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

given: , polyhedron

promise:


finding the next ellipsoidsuppose we already computed ellipsoids

R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

E ⊃ ⋯ ⊃ E ⊇ P0 k

given: , polyhedron

promise:



if center of satisfies , we can stop (set )

R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

E ⊃ ⋯ ⊃ E ⊇ P0 k

sk Ek {Ax ⩽ b} N := k

given: , polyhedron

promise:




otherwise, violates a constraint in

R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

E ⊃ ⋯ ⊃ E ⊇ P0 k


sk {a x ⩽ b }i⊺

i {Ax ⩽ b}

given: , polyhedron

promise:





in this case, we let be the minimum-volume ellipsoid thatcontains half-ellipsoid

R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

E ⊃ ⋯ ⊃ E ⊇ P0 k


sk {a x ⩽ b }i⊺

i {Ax ⩽ b}

Ek+1

E ∩ {x ∣ a (x − s ) ⩽ 0} ⊇ Pk i⊺

k

given: , polyhedron

promise:






remains to prove:

R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

E ⊃ ⋯ ⊃ E ⊇ P0 k


sk {a x ⩽ b }i⊺

i {Ax ⩽ b}

Ek+1

E ∩ {x ∣ a (x − s ) ⩽ 0} ⊇ Pk i⊺

k

volE ⩽ e ⋅ volEk+1−1/n

k

given: , polyhedron

promise:






remains to prove:

also remains to prove (but will skip): no numerical issues

R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

E ⊃ ⋯ ⊃ E ⊇ P0 k


sk {a x ⩽ b }i⊺

i {Ax ⩽ b}

Ek+1

E ∩ {x ∣ a (x − s ) ⩽ 0} ⊇ Pk i⊺

k

volE ⩽ e ⋅ volEk+1−1/n

k

geometry of ellpsoids


theorem: let be an ellipsoid with center let be a halfspace with on its boundarythen, ellipsoid that contains such that

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)



what’s an ellipsoid?

any non-singular affine transformation of

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

x ↦ Mx + s

unit ball E = Ball(0, 1) = {x ∣ x x ⩽ 1}0⊺





what the volume of an ellipsoid?

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

x ↦ Mx + s







an affine transformation changes any volume bya multiplicative factor of

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

x ↦ Mx + s


x ↦ Mx + s

∣detM ∣




therefore, if

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

x ↦ Mx + s

∣detM ∣

volE = ∣detM ∣ ⋅ volE0 E = M(E ) + s0




therefore, if

strategy for proof of theorem:

first prove unit-ball case then extent to general case by “abstract nonsense”

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

x ↦ Mx + s

∣detM ∣

volE = ∣detM ∣ ⋅ volE0 E = M(E ) + s0

E = E0


unit-ball case

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

E = E = {x ∣ x x ⩽ 1}0⊺


unit-ball case

let be a halfspace with on its boundary

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

E = E = {x ∣ x x ⩽ 1}0⊺

H0 0


unit-ball case


by rotational symmetry, we can assume

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

E = E = {x ∣ x x ⩽ 1}0⊺

H0 0

H = {x ∣ x ⩾ 0}0 1


unit-ball case



let , where

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

E = E = {x ∣ x x ⩽ 1}0⊺

H0 0

H = {x ∣ x ⩾ 0}0 1

E = M(E ) + t ⋅ e0′

0 1 M = diag(a, b,… , b)


unit-ball case



let , where

claim: if and , then

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

E = E = {x ∣ x x ⩽ 1}0⊺

H0 0

H = {x ∣ x ⩾ 0}0 1

E = M(E ) + t ⋅ e0′

0 1 M = diag(a, b,… , b)

⩽ 1a2

(1−t)2 + ⩽ 1a2t2

b21 E ⊇ E ∩ H0

′0 0


unit-ball case



let , where


note

the conditions ensure …

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

E = E = {x ∣ x x ⩽ 1}0⊺

H0 0

H = {x ∣ x ⩾ 0}0 1

E = M(E ) + t ⋅ e0′

0 1 M = diag(a, b,… , b)

⩽ 1a2

(1−t)2 + ⩽ 1a2t2

b21 E ⊇ E ∩ H0

′0 0

E = {y ∣ (y − t ⋅ e ) M (y − t ⋅ e ) ⩽ 1}0′

1⊺ −2

1

e , e ∈ E1 2 0′


unit-ball case



let , where


claim: minimum value of subject to these conditions satisfies

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

E = E = {x ∣ x x ⩽ 1}0⊺

H0 0

H = {x ∣ x ⩾ 0}0 1

E = M(E ) + t ⋅ e0′

0 1 M = diag(a, b,… , b)

⩽ 1a2

(1−t)2 + ⩽ 1a2t2

b21 E ⊇ E ∩ H0

′0 0

detM = abn−1

⩽ e−1/(2n+2)


general case

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

E = T (E ) + s0


general case

then, is halfspace with on its boundary

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

E = T (E ) + s0

H = T (H − s)0−1 0


general case


let be minimum-volume ellpsoid containing

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

E = T (E ) + s0

H = T (H − s)0−1 0

E0′ E ∩ H0 0


general case



we know (from unit-ball case)

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

E = T (E ) + s0

H = T (H − s)0−1 0

E0′ E ∩ H0 0

volE ⩽ e volE0′ −1/(2n+2)

0


general case




choose

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

E = T (E ) + s0

H = T (H − s)0−1 0

E0′ E ∩ H0 0


0

E = T (E ) + s′0′


general case




choose

then

E ⊆ Rn s

H ⊆ Rn s

∃ E ⊆ R′ n E ∩ H

volE ⩽ e ⋅ volE′ −1/(2n+2)

E = T (E ) + s0

H = T (H − s)0−1 0

E0′ E ∩ H0 0


0

E = T (E ) + s′0′

= ⩽ evol(E)vol(E )′

vol(E )0

vol(E )0′

−1/(2n+2)

feasibility: relaxed vs non-relaxedlast time: poly-time algorithm for relaxed feasibility problem

given: , polyhedron

promise:


R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P


given: , polyhedron

promise:


today: turn it into algorithm for general linear programming

R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P


given: , polyhedron

promise:


today: turn it into algorithm for general linear programming

first step: algorithm for the (non-relaxed) feasibility problem

given: system of linear inequalities

goal: decide if system has a solution or not

R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

{Ax ⩽ b}




strategy: compute new system such that

if feasible, then satisfies relaxedfeasibility promise for with

�.

if infeasible, then infeasible�.

last time: poly-time algorithm for relaxed feasibility problem

given: , polyhedron

promise:


R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

{Ax ⩽ b}

{ x ⩽ }A~

b~

{Ax ⩽ b} { x ⩽ }A~

b~

R, ε ⟨R⟩ + ⟨ε⟩ ⩽ poly(⟨A⟩, ⟨b⟩){Ax ⩽ b} { x ⩽ }A

~b~






�.


it follows that the relaxed-feasibility algorithm finds solution to if and only if is feasible


given: , polyhedron

promise:


R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

{Ax ⩽ b}

{ x ⩽ }A~

b~

{Ax ⩽ b} { x ⩽ }A~

b~


~b~

{ x ⩽ }A~

b~

{Ax ⩽ b}






�.


it follows that the relaxed-feasibility algorithm finds solution to if and only if is feasible can decide

feasibility of in polynomial time


given: , polyhedron

promise:


R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

{Ax ⩽ b}

{ x ⩽ }A~

b~

{Ax ⩽ b} { x ⩽ }A~

b~


~b~

{ x ⩽ }A~

b~

{Ax ⩽ b} ⇝{Ax ⩽ b}






�.


remains to show: construction of


given: , polyhedron

promise:


R > ε > 0 P = {x ∈ R ∣ Ax ⩽ b}n


y ∈ P

{Ax ⩽ b}

{ x ⩽ }A~

b~

{Ax ⩽ b} { x ⩽ }A~

b~


~b~

{ x ⩽ }A~

b~

not-too-large promise

lemma: feasible if and only if the followingsystem is feasible for ,

lemma follows from the fact that feasible systems of linearinequalities always have solutions with small encoding size

{Ax ⩽ b}R = 100⟨A⟩+⟨b⟩

{Ax ⩽ b, −R ⋅ 1 ⩽ x ⩽ R ⋅ 1}

not-too-small promise


let be a system of linear inequalities

for , let be set of solutions of

{Ax ⩽ b}

η ⩾ 0 Pη {Ax ⩽ b + η ⋅ 1}




lemma: with and

if , then for some �. if , then �.

{Ax ⩽ b}

η ⩾ 0 Pη {Ax ⩽ b + η ⋅ 1}

∃η, ε > 0 ⟨η⟩ + ⟨ε⟩ ⩽ poly(⟨A⟩, ⟨b⟩)

P ≠ ∅0 Ball(y, ε) ⊆ Pη y ∈ Rn

P = ∅0 P = ∅η




lemma: with and


proof of 1: let

{Ax ⩽ b}

η ⩾ 0 Pη {Ax ⩽ b + η ⋅ 1}

∃η, ε > 0 ⟨η⟩ + ⟨ε⟩ ⩽ poly(⟨A⟩, ⟨b⟩)


P = ∅0 P = ∅η

y ∈ P0




lemma: with and


proof of 1: let

then,

{Ax ⩽ b}

η ⩾ 0 Pη {Ax ⩽ b + η ⋅ 1}

∃η, ε > 0 ⟨η⟩ + ⟨ε⟩ ⩽ poly(⟨A⟩, ⟨b⟩)


P = ∅0 P = ∅η

y ∈ P0

Ax = Ay + A(x − y) ⩽ b + ∥A(x − y)∥ ⋅ 1∞




lemma: with and


proof of 1: let

then,

also

{Ax ⩽ b}

η ⩾ 0 Pη {Ax ⩽ b + η ⋅ 1}

∃η, ε > 0 ⟨η⟩ + ⟨ε⟩ ⩽ poly(⟨A⟩, ⟨b⟩)


P = ∅0 P = ∅η

y ∈ P0

Ax = Ay + A(x − y) ⩽ b + ∥A(x − y)∥ ⋅ 1∞

∥A(x − y)∥ ⩽ 2 ∥x − y∥∞⟨A⟩

2




lemma: with and


proof of 1: let

then,

also

therefore, whenever

{Ax ⩽ b}

η ⩾ 0 Pη {Ax ⩽ b + η ⋅ 1}

∃η, ε > 0 ⟨η⟩ + ⟨ε⟩ ⩽ poly(⟨A⟩, ⟨b⟩)


P = ∅0 P = ∅η

y ∈ P0

Ax = Ay + A(x − y) ⩽ b + ∥A(x − y)∥ ⋅ 1∞

∥A(x − y)∥ ⩽ 2 ∥x − y∥∞⟨A⟩

2

Ax ⩽ b + η ⋅ 1 ∥x − y∥ ⩽ η ⋅ 22−⟨A⟩ □




lemma: with and


proof of 2:

{Ax ⩽ b}

η ⩾ 0 Pη {Ax ⩽ b + η ⋅ 1}

∃η, ε > 0 ⟨η⟩ + ⟨ε⟩ ⩽ poly(⟨A⟩, ⟨b⟩)


P = ∅0 P = ∅η




lemma: with and


proof of 2: surprising that this part of lemma is true

strategy: argue about infeasibility certificates

{Ax ⩽ b}

η ⩾ 0 Pη {Ax ⩽ b + η ⋅ 1}

∃η, ε > 0 ⟨η⟩ + ⟨ε⟩ ⩽ poly(⟨A⟩, ⟨b⟩)


P = ∅0 P = ∅η




lemma: with and




if , then by Farkas lemma with and; we may also assume

{Ax ⩽ b}

η ⩾ 0 Pη {Ax ⩽ b + η ⋅ 1}

∃η, ε > 0 ⟨η⟩ + ⟨ε⟩ ⩽ poly(⟨A⟩, ⟨b⟩)


P = ∅0 P = ∅η

P = ∅ ∃y ⩾ 0 A y = 0⊺

b y = −1⊺ ⟨y⟩ ⩽ poly(⟨A⟩, ⟨b⟩)




lemma: with and





claim: also certifies

{Ax ⩽ b}

η ⩾ 0 Pη {Ax ⩽ b + η ⋅ 1}

∃η, ε > 0 ⟨η⟩ + ⟨ε⟩ ⩽ poly(⟨A⟩, ⟨b⟩)


P = ∅0 P = ∅η

P = ∅ ∃y ⩾ 0 A y = 0⊺

b y = −1⊺ ⟨y⟩ ⩽ poly(⟨A⟩, ⟨b⟩)

y P = ∅η




lemma: with and





claim: also certifies

proof: for small enough

{Ax ⩽ b}

η ⩾ 0 Pη {Ax ⩽ b + η ⋅ 1}

∃η, ε > 0 ⟨η⟩ + ⟨ε⟩ ⩽ poly(⟨A⟩, ⟨b⟩)


P = ∅0 P = ∅η

P = ∅ ∃y ⩾ 0 A y = 0⊺

b y = −1⊺ ⟨y⟩ ⩽ poly(⟨A⟩, ⟨b⟩)

y P = ∅η

(b + η ⋅ 1) y ⩽ −1 + η ⋅ 2 < 0⊺ O(⟨y⟩) η

decision vs searchso far: we saw a poly-time algorithm for deciding thefeasibility of systems of linear inequalities


next: if solution exists, can also find one in poly time



idea: look for basic feasible solution because it is enough toknow the list of tight constraints for such a solution




strategy: given a system , find -maximalcoordinate set such that remains feasible

{Ax = b,x ⩾ 0} ⊆S {Ax = b, x ⩾ 0, x = 0}S





can find such a set by starting with and growing oneby one using the algorithm for deciding feasibility

{Ax = b,x ⩾ 0} ⊆S {Ax = b, x ⩾ 0, x = 0}S

S = ∅ S





can find such a set by starting with and growing oneby one using the algorithm for deciding feasibility

turns out: if we found such a set, the linear equations uniquely determine a feasible solution

for the given system

{Ax = b,x ⩾ 0} ⊆S {Ax = b, x ⩾ 0, x = 0}S

S = ∅ S

{Ax = b,x = 0}S x

feasible vs optimal solutions

last step: turn our algorithm for finding a feasible solution to alinear program into algorithm for finding an optimal solution




good general strategy: use binary search to find the largest such that is feasible


γ

{c x ⩾ γ,Ax ⩽ b,x ⩾ 0}⊺




shortcut for linear programs: use strong duality; find“matching pair” of feasible solutions to (P) and its dual (D)


γ

{c x ⩾ γ,Ax ⩽ b,x ⩾ 0}⊺





shortcut for linear programs: use strong duality; find“matching pair” of feasible solutions to (P) and its dual (D)

concretely, find feasible solutions and for the followingsystem—guaranteed to be optimal solutions for (P) and (D)


γ

{c x ⩾ γ,Ax ⩽ b,x ⩾ 0}⊺


x y

{c x = b y, Ax ⩽ b,x ⩾ 0, A y ⩾ c, y ⩾ 0}⊺ ⊺ ⊺

linear programming - ETH Z · theorem: every feasible, bounded linear program in equational form...

Documents

Transcript of linear programming - ETH Z · theorem: every feasible, bounded linear program in equational form...