Linear Programming

Reading: CLRS, Ch. 29 or reference

CSE 6331 Algorithms

Steve Lai

1 2

1 2

1 2 1 1 2 2

Let , , , , be real numbers and

let , , , be variables.

Linear function:

( , , , )

Linear equality

Linear functions and linear constraints

n

n

n n n

a a a b

x x x

f x x x a x a x a x

1 2

1 2

1 2

:

( , , , ) (hyperplane)

Linear inequalities:

( , , , ) (half-space)

( , , , ) (half-space)

n

n

n

f x x x b

f x x x b

f x x x b

2

1 2

1 1 1 2

Given real numbers , , , where 1 , 1 ,

find real numbers , , , to

optimize a linear objective function

General Linear Programming Problems

i j ij

n

n n

b c a i m j n

x x x

c x c x c x

1

linear constraints

s

ubject

for 1 .

to

n

ij j i

j

a x b i m

3

1 2

1 1 2 2

1

find real rnu

1 , 1

1

1

, ,

Given r

, mbers

eal numbers

that

for 1

0 f

m

aximize

subject to

o

Standard Maximum Form

ij

i

j

n

n n

n

ij j i

j

j

a i m j n

b i m

c j n

x x x

c x c x c x

a x b i m

x

r 1 j n

4

1 1 2 2

11 12 1 1 1

21 22 2 2 2

1 1

( )

and

See t

maximize

subject o

he

t

In Matrix Notation

T

n n

n

n

m m mn n n

c x c x c x

a a a x b

a a a x b

a a a x b

c x

Ax b x 0

next slide for notation.

5

1 2 1 2

1 2 1 2

11 12 1

21 22 2

1 1

, , , , , , ,

, , , , ,

-vectors:

, ,-vectors:

-matri

x

:

Notation

T T

n n

T T

m m

n

n

m m mn

x x x c c c

b b b y y

n

m y

a a a

a a a

a a

m n

a

x c

b y

A

6

1 1 2 2

11 12 1

21 22 2

1 2 1 2

1 1

minimize

sub

( )

ajec ndt to

Standard Minimum Form (in Matrix Notation)

T

m m

T T

n

n

m n

m m mn

y b y b y b

a a a

a a ay y y c c c

a a a

y b

y A c y 0

7

1 2

1 2

Variables: , , ,

Feasible solution: a vector ( , , , ) that

satisfies all the constraints.

Feasible region : the set of all feasible solutions.

Optimal solut

Terminology

n

n

x x x

x x x

ion: a feasible solution that optimizes the

objective function.

Optimal objective value *:

A linear program may be

( is nonefeasible infeasiblempty) or ( is empty)

(fea

sible)

f

b or ( * is finite or infinite)ounded unbounded f

8

1 2

1 2

1 2

1 2

1 2

maximize

subject to

2 4

4 2 12

1

, 0

objective function

main constraint

s

Example

x x

x x

x x

x x

x x

nonnegativity const ra ints

9

10 Source: Linear programming, Thomas S.

Ferguson, UCLA

Two maximization LPs, and , are if

for each feasible solution to with objective value

there is a

equivalent

correspond feasible solution ting o with

Equivalence of Linear Programs

L L

zL

L

x

x

equivalent

objective value , and .

A maximization LP, , and a minimization LP, ,

are if for each feasible solution to with

objective value there is a cor

vi

re

ce

sponding f

ve

ea

rsa

sibl

z

z

L L

L

x

e

solution to with objective value , and

vice versa.

L z x

11

We can always convert an LP into an LP in

standard maximum (or minimum) form.

A linear program may not

equi

be

valen

in standard maximum form

for any of thes

t

Converting to Standard Maximum Form

e reasons:

1. The objective funtion may be a minimization.

2. There may be " " (instaed of " ") inequality constraints.

3. There may be equality constraints.

4. There may be some without nonnegix

ativity constraints.

12

1 1

1 1

1. Convert minimization to maximixation:

Minimize ( , , ) maximize ( , , )

2. Convert a " " constraint to a " " constraint:

( )

3. Convert an equality c

n n

n n

ij j i ij j i

j j

f x x f x x

a x b a x b

1 1

1

1 1

onstraint into two inequalities:

( )

n n

ij j i ij j in

j j

ij j i n nj

ij j i ij j i

j j

a x b a x b

a x b

a x b a x b

13

1

4. If is without nonnegativity constraint,

we replace with and add 0, 0.

Thus, becomes , and

becomes .

A feasible solution ,

j

j j j j j

j j j j j j

j j j j j j

x

x x x x x

c x c x c x

a x a x a x

x

1

, , , , to the new LP

corresponds to a feasible solution , , , ,

to the original LP, where .

j j n

j n

j j j

x x x

x x x

x x x

14

1 2 1 2

1 2

1 2

1 2

1 2

1 2

1

1

2 3 2 3

subject to

min

subject

im

to

77

7 2 4

ize maximiz

2 4

0

e

0

Example

x x x x

x xx x

x xx x

x xx

x

15

1 1

2 2 2

2

1 2 1 2 2

1 2 1 2

1

2

1 2

2maximize 2 maximize 2

subject to

Replace wit

subject to

7 7

7

2 4

h

0

:

3 3 3

x x x

x x xx x

x x x x x

x x x x

x x

x

2

1 2 2

1 2 2

7

2 2 4

, , 0

x

x x x

x x x

16

1 2 3

1 2 3

1 2 3

1 2 3

2 3

3

2 2

1 2

maximize 2 3 3

subject to

7

7

Renaming ,

2 2

as , yie

4

, , 0

lds:x

x x x

x x x

x x x

x x x

x x x

x x x

17

1 1 2 2

Each LP has an intimately related LP, called its dual.

The original LP is called the pri

maximi

mal.

Primal (standar

ze

subjec

d maximum )

t

L :

P

Duality

T

n nc x c x c x

c x

1 1 2 2

to

minimiz

and

e

Dua

subject

l (stan

andto

dard minimum LP

):

T

m m

T T

y b y b y b

Ax b x 0

y b

y A c y 0

18

1 2

1 11 12 1 1

2 21 22 2

The dual of the Standard Minimum LP is the

Standard Maximum LP.

The primal max LP and the dual min LP can be

simultaneously displayed as:

n

n

n

x x x

y a a a b

y a a a

2

1 2

1 2

m m m mn m

n

b

y a a a b

c c c

19

1 2 1 2 3

1 2

1 2

1 2

maximize minimize 4 12

subject to

Primal problem: Dual

subject to

2 4

problem

4 2 12

:

Example

x x y y y

x x

x x

x x

1 2 3

1 2 3

1 2 1 2 3

4 1

2 2 1

1

, 0 , , 0

y y y

y y y

x x y y y

20

1 2

1

2

3

1 2 4

4 2 12

1 1 1

1 1

Example

x x

y

y

y

21

If is a feasible solution to the standard maximum

LP ( , , ) and a feasible solution to its dual, then

Theorem.

Proof.

.

,

,

Weak Duality Theorem

T T

T T

T T T T

x

A b c y c x y b

Ax b x 0 y Ax y b

y A c y 0 y Ax c x

Corollary.

Corollar

.

If a standard LP and its dual are both feasible,

then they are bounded.

If and are feasible solutions to the primal and

dual LPs, respectively, and

y

,

.

then

T T

T T

c x y b

x y

c x y b

x and are optimal solutions to their respective problems.y

22

A feasible solution to the primal linear program

( , , ) is optimal iff there is a feasible solution to its dual

linear program such

Theore

that . In this ca

m.

se, is

Strong Duality Theorem

T T

x

A b c y

c x y b y also

optimal.

The primal LP has an optimal solution iff its dual

has an optimal solution, in which case, their optimal values

a

Corolla

re e

ry:

qual.

23

Primal

feasible bounded feasible unbounded infeasible

f.b. yes no noDual

f.u. no no yes

i no yes yes

24

1 2 1 2

1 2

11 1 1

wh

to

er

Find

maximize

and

, , , , , , ,

, , ,

+

subje

t o

e

c t

LP in equality form

T

T T

n m m

T

n m

n n

x x x b b b

c c c

a x a x

x

c x

Ax b x 0

x b

c

1, 1

21 1 2 2, 2

1 1 ,

+

+

n m n m

n n n m n m

m mn n m n m n m m

a x b

a x a x a x b

a x a x a x b

25

Assume rank( ) .

Let be a non-singular matrix formed by choosing

columns out of the columns of .

The variables associated with the columns in are

calle

bd

Basic Solutions

i

A m

B m

m n A

m x B

the other variables are

.

If all the non-basic variables are set to zero, then the

solution to the resulting s

asic variables; non-basic

ystem of equations is calle

variable

d a

s

ba so

sic

n

n

lution

basic feasible solu

.

If a basic solution is feasible, it is a .

( )! The number of basic solutions is at most .

tion

! !

m n

m

m nC

m n

26

1 2

1 2

A set is said to be if for all pointsconvex

extreme poin

,

we have (1 ) for all 0 1.

A point in a convex set is called an

t

Basic feasible solutions are extreme points

C x x C

x x C

x C

1 2 1 2

1 2

if there exist no , , , and 0 1,

such that (1 ) .

The feasible region of

Basic feasible solutions

an LP is a convex set.

of an LP (in the equality form

extr

)

are eme po

x x C x x

x x x

of the feasible region.

If the LP is feasible bounded, th

ints

optimum sol

en at least one

occurs at an extreme put oio .n int

27

Simplex Method

A system of linear

inequalities defines a

polytope (or simplex) as

a feasible region.

The simplex algorithm

begins at a starting

vertex and moves along

the edges of the polytope

until it reaches the vertex

of the optimum solution.

28 Source: Linear programming, Thomas S.

Ferguson, UCLA

v̂

1 2

1 2

1 2

1 2

1 2

maximize

subject to

2 4

4 2 12

1

, 0

objective function

main constraint

s

Example

x x

x x

x x

x x

x x

nonnegativity const ra ints

29

1 2 1 2

Standard max form Slack form

maximize maximize

subject to subject t

Example

x x x x

1 2 1 2 1

1 2 1 2 2

1 2 1 2 3

1 2 1 2 1 2 3

o

2 4 2 4

4 2 12 4 2 12

1 1

, 0 , , , , 0

x x x x s

x x x x s

x x x x s

x x x x s s s

30

31 Source: Linear programming, Thomas S.

Ferguson, UCLA

1 2

Find

maximize

to

subject t and

and ( , , , ) to

Standard maximum form:

Equivalent sl

o

Find

ac

m

k for :

Converting standard form to slack form

T

T

ms s s

x

c x

Ax b x 0

x s

(or minimize )

and ,

may be written a

maximize

subjec

t to

s .or

T T

c x c x

Ax s b x 0 s

Ax s b s b Ax s Ax b

0

32

1 2

1 2

, , , are called .

The equality and the objective

function can be displayed in a tableau:

(to m

slack var

inimize

iable

, where

1

0)

s

T

m

n

Tz v

z

s s s

x

v v

x x

s Ax b

c x

c x

1 11 12 1 1

2 21 22 2 2

1 2

1

1

n

n

m m m mn m

m n

s a a a b

s a a a b

s a a a b

z c c c v

33

1 2

1

( ) (or minimize )

and , :

Initial basic variables: , , ,

Initial non-basi

maximize

subject to

c variables:

Basic and non-basic variables

T T

m

f

B s s s

N x

x c x c x

s Ax b x 0 s 0

2

( , ) basic solution.

( , ) basic feasible s

, , ,

The solution is a

If , is a .

If in , , the solution

olution

( is .

All feasible sol

, ) optimu

utions

m

( ,

)

nx x

b 0

b 0

x 0 s b

x 0 s b

x 0 s b

x

c 0

s have objective value ( ) 0.

The solution has objective value (( , ) 0) .

f

f

x b x0 s

x

34

35

1

1 11 1 1 1

1

1

1

s.t. and , :

1 1

where 0 initially

If

Max

0, we can pivot ar

imi e

u

z

o

Pivot OperationT

n

j n

i in i

m m mj mn m

j n

j

ii j

i

jx

s

z v

x x

s a a a b

a a b

s a a a b

z c c c v v

a

a

c x s Ax b x 0 s 0

nd , switching the roles

of and

ij

i j

a

s x

36

1 1 , 1 1 , 1 1

, 1 , 111 1 1

Dividing both sides of the equation

by , we have an equivalent equation:

ij

ij

ij ij ij ij ij i

i i j j i j j in n i

i j i ji in ij j

j

n

i j

ij

a x a x a x a x b

a aa a bx x x

a

a

a a a a ax

sx

a

s x

1

1 1

.

Substituting this into every other equation yields:

kj i kj in

k k kn n

ij ij

kj i

k

ij

kj

i

ij

a a a as a x a x

a a

as

a

a bb

a

37

1 1

1

1 1

Also, substitute it into

( ) ( )

and we have:

( )( )

( ) ( )

( )

T

n n

j i

ij

j in

n

j

i

ij

ij

z v

c x c x

c

v

c az c x

a

c

s

c

a

a

a

c x

( )j i

n

ij

c bx v

a

38

1

1 1

1 11 1 1

1

1

1

1

After pivoting around , we have the equivalent LP:

1 1

ˆ

ˆ

ˆ ˆ ˆ ˆ

1

n

j in j i

n

ij ij

mj in mj i

m m m

i

j

ij

i in ij

i

n m

i

j

ij

ij ij

mj

ij

j ij

j n

ij

x x

a a a

s

a

a

a a bx

bs a a b

a a

a a a bs a a b

a a

z c c c v

a

a

a

a a

a

a

39

1

1. Pivot element: 1

2. Other elements in the pivot row:

3. Other elements in the pivot column:

4. All other elements

E xample:

:

3

Summary of Pivot Operation

r p r p

c q c p q rc p

p p

r r p

c c p

q q rc

p

p

0 5 3 2 9 2 7 2

3 1 1 2 3 2 1 2

0 4 3 0 4 3

2

1 1

Find

minimize

subject

to

an

Standard minimum

form:

Equiva

to

Find

d

and ( , , , )

lent slack fo m:

t

r

Converting minimum form to slack form

T

T T

T

ns s s

y

y b

y A c y 0

y s

minimize

subje

o

and ct o t ,

T

T T T

y b

s y A c y 0 s 0

40

1 2

1 11 12 1 1

2 21 22 2 2 1 2

1 2

1 2

1

The equality and objective function

can be displayed in a tableau:

, , ,

, , ,

0

T e h

n

n

n m

n

m m m mn m

m

T T T T

n

s s s

y a a a b

y a a a b B s s s

N y y y

y a a a b

c c c

s y A c y b

, basic solution.

basic feasible soluti

solution is a

If , the solution is a .

If and , the solution i

o

s a

n

optimum solut onn i

y 0 s c

c 0

b 0 c 041

42

1

So we display both the primal and the dual in the

same tableau.

Pivoting for the dual follows the same rule:

r p r p

c q c p q rc p

p

1 2

1 11 12 1 1

2 21 22 2 2

1 2

1

0

n

n

n

m m m mn m

m n

x x x

y a a a b

y a a a b

y a a a b

z c c c

43

1 2

1 11 12 1 1

2 21 22 2 2

Pivot madly until we suddently find that all entries in the last

row and last column (exclusive of the corner) are nonnegative.

ˆˆ ˆ ˆ

ˆˆ ˆ ˆ

ˆ

The Pivot Madly Method

n

n

n

m

r r r

t a a a b

t a a a b

t

1

1

1

1 2

1

1

0 for , ,

ˆ for , ,

ˆ ˆ for , ,ˆˆ ˆ

0 fo

Solution for the primal:

Solution for the dual:

ˆOp

r , , 0ˆ

timal valueˆ ˆ ˆ

j j n

i i i m

j j n j

m m mn m

i i m

m n

x x r r

x b x t t

y c r r ca a a b

y y t tc c c

vv

44

1 2 3

1

1

2 32

4

1 43

2 3

01

4,

Max solution for primal:

Min solution for

34

1

dual:

Optimal value 1

0 03

2, 45 2 4 16

6

Example

x y yx

yx x

x

yy y

xy y

45

Suppose after pivoting for a while we have the tableau:

If , then is a feasible solution for the

maximization problem.

If , then is a f

,

, easible s

v

r 0 t b

t 0

r

t A b

c

r0 c

b 0

c olution for the

minimization problem.

If and , then we have an optimal solution

for both problems.

b 0 c 0

46

The same as the pivot madly method except that we now

choose the point elements more systematically.

The goal is to make all entries in the last row and last

column (exclus

The Simplex Method

ive of the corner) nonnegative.

v

r

t A b

c

47

0

0

0 0

0

, 0

, ,

Case 1: . Take any column with 0. Among

those with 0, choose to be the for which the ratio

is smallest. Pivot around

Pivot rules for the simplex method (1)

j

i j

i i j i

j c

i a i i

b a a

b 0

0 0

1 2 3 4 5 1 2 3

1 1

2 2

3 3

4 4

. (If no such ,

the max problem is feasible unbounded.)

1 1 1 6 6 1 3 1

0 4 4 4 0 2 4

0 3 1 7

3

3 2

2

3

7 2 3 1

5 1 4 2 0 5 4 2

5 2 4 2 1 0 5 4 0

3

1

2

1

2

j i

r r r r r r r r

t t

t t

t t

t t

48

49

0

0

0

0

0

0

0

,

,

After the pivot, stays nonnegative, the entry

becomes positive, and the value never gets less

( gets greater if 0).

1

Properties of rule 1:

j

i

i

i j i

j

i j

c

v

v b

r p r p

c q c p q rc p

b

a b

c

a

v

p

b

0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

, ,

, , , ,

, ,

1

i j i i j

i j i j i i j i i j

j i j j i i j

a b a

a a b a b a

c a v c b a

0, 0

Case 2: some are negative. Take the first 0. Find any

negative entry in row , say 0. (If there is no such ,

the max problem is infeasible.)

Pivot rules for the simplex method (2)

i k

k j

b b

k a j

0 0

0 0 0 0 0

0 0

, ,

, 0 ,

0 ,

1 2 3 4

1

2

3

4

Compare with

for which 0, 0 and choose such that

is smallest ( may be .) Pivot around .

1 3 1

1 2 4

1 2

4

5

4

2 1 3 1 2

1 4 2 0

5 2 4 2 0

k k j i i j

i i j i i j

i j

b a b a

b a i b a

i k a

r r r r

t

t

t

t

1 2 3

1

2

3

4

0 1 2 4

1 3 3 1

2 2 3 1

5 1 4 0

r r r

t

t

t

t

50

51

0

The objective in Case 2 is to get to Case 1.

With the pivot, the nonnegative stay nonnegative,

and becomes no smaller (it gets larger if 0).

1

Properties of rule 2:

i

k i

b

b b

r p r p

c q c c

p

p q r p

0 0 0 0 00

0 0 0 0 0 0 0 0

00 0 0 0 0 0 0

0 0, ,

, , , ,

, ,

,

,

1

i j i i ji

i j i i j i j i i j i i j

k j kk j i j

i j

k k j i i j

a b ab

a b a a b a b a

a

a

b a a b a b a

0

0 0

0

, 0 ,

Case 3: . Treat it as the dual problem.

Take any such that 0. Among those with

0, choose

row

the for which the ratio is

Pivot rules for the simplex method (3)

i

i j j i j

i b j

a j c a

c 0

0 0, 0

1 2 3 4

1

2

3

closest to zero. Pivot around . (If there is no such ,

the dual problem is feasible unbounded.)

1 3 1 6

0 3 2 4 4

2 0 3 1 2

5 1 4 2 3

1

i ja j

r r r r

t

t

t

52

1 2 3

2 3

1 3

1 2 3

1 2 3

2 3

1

Primal: maximize 2 subject to all 0 and

2 3

3 2

2 1

Dual: minimize 3 2 subject to all 0 and

2 1

Example

i

i

x x x x

x x

x x

x x x

y y y y

y y

y y

3

1 2 3

1

2 3 2y y y

53

1 2 3 1 3 3

1 1

2 2

3 2

1 3 2

1 1

3

2

0 1 2 3 2 1 1 2

1 0 3 2 1 0 2

2 1 1 2 1 1 1

1 1 2 0 1 1 1 1

5 3 1 1 3 4 3

1 3 0 1 3 2 3

7 3 1 1 3 1 3

Maximum solution for

2 3 1 1 3 5 3

primal:

3

1

x x x x y x

y y

y y

y x

x y y

y x

x

x

2 3

1 2 3

0, 1 3, 2 3

l

0,

Minimum solution for dua

Optimal value for bot

1 3

h:

, 1

5 3

x x

y y y

54

The simplex rules as stated may lead to cycling,

even though it is rare in practical problems.

There are strategies (modified simplex methods) for

avoiding

Sm

cycling.

allest-subscript r

Cycling

If there is a choice of pivot rows

or a choice of pivot columns, select the row (or column)

with the variable having the lowest subscript, or if there

are no variables, with the

u

varia

le:

x

x y ble having the lowest

subscript.

55

Formulating problems as linear programs

Problem: Given a weighted directed graph ( , ) and

two vertice find the shortest dists , , .

For each vertex , let denote the shortest di

ance from

st

t

n

o

a

Single-pair shortest path

v

G V E

s t s tV

v V d

ce

from to . (We want to find .)

The values satisfy the following constraints:

0 and for each edge ( , ) , .

If values satisfy the above constraints, then .

( , )

t

v

s

v

v u

t t

d d

s v d

d

d u v wE

x x d

u v

Thus, we obtain the following LP (with variables ):

Maximize

Subject to ( , ) for each edge ( , )

0 and 0 for all { }.

v

t

v u

s v

x v V

x

x x w u v u v E

x x v V s

57

( , )

For each edge ( , ) , introduce a variable .

Minimize // is the weight of ( , )//

Subject to

1 if

1 if

0

An alternative formulation

ij

ij ij ij

i j E

ij ji

j j

i j E x

x w w i j

i s

x x i t

Every basic optimal solution (when exists) ha

for all ;

otherwise

and

s

all variables equal to 0 or 1,

a

0 f

nd

or a

the set of edges whose

variab

ll ( , ) .

Theorem:

les eq

ij

i V

x i j E

ual 1 form an - directed path.s t58

( , ), capacity function , source , sink .

A flow is a real-valued function : satisfying

Capacity constraint: , , ( , ) ( , ).

Skew symmetry: , , ( ,

Maximum Flows

G V E c s t

f V V

u v V f u v c u v

u v V f u

) ( , ).

Flow conservation: { , },

( , ) ( , ) 0

The value of a flow is ( , ) ( , ).

The maxflow problem is to find a maximum flow.

v V

v V

v f v u

u V s t

f u V f u v

f f s v f s V

59

References

• T.S. Ferguson, “Linear Programming: A

Concise Introduction,”

http://www.math.ucla.edu/~tom/LP.pdf .

• Lecture notes:

http://www.math.cuhk.edu.hk/~wei/LP11.html

60