Linear Programming Relaxation

Department of Computer Science

and Information Engineering

Linear Programming Relaxation

1

Instructor: Yao-Ting Huang

Bioinformatics Laboratory,

Department of Computer Science & Information Engineering,

National Chung Cheng University.

Dept. of Computer Science

& Information Engineering

Problem Solving Strategies

NP-hard problems can be solved by many methods. Branch and bound (B&B): a general solution for finding

optimal solutions of various optimization problems.

Meta-heuristic search, e.g., Genetic Algorithm

Statistical learning approach: Expectation Maximization, Gibbs sampling, …

Fixed-parameter algorithm.

Approximation algorithm. Deterministic

Randomized

Relaxation

Polynomial-Time Approximation Scheme

2



Integer/Linear Programming

An integer (linear) -programming problem is the

problem of either minimizing or maximizing an

integer (linear) function subject to a finite set of

integer (linear) constraints.

If the objective is to minimize, then we call the problem is a

minimization integer (linear) program.

If the objective is to maximize, then we call the problem is

a maximization integer (linear) program.

Integer programming is NP-hard.

3



An Example of Integer Programming

Maximize

x1 + x2 (29.11)

subject to

4x1 – x2 8 (29.12)

2x1 + x2 10 (29.13)

5x1 – 2x2 -2 (29.14)

x1, x2 = 1 or 0. (29.15)

4Constraints

Objective Function



Space of Feasible Solutions

5

https://www.google.com.tw/imgres?imgurl=http://glassproperties.com/optimization/non-

convex_optimization.gif&imgrefurl=http://www.snipview.com/q/Convex%2520optimization&h=384&w=512&tbni

d=kAYmrhtx8IpcUM:&docid=7wx__AWoEDWEbM&hl=zh-

TW&ei=HuhoVvbnOsO8mgXL3K2QCQ&tbm=isch&ved=0ahUKEwj27on_o9DJAhVDnqYKHUtuC5IQMwgiKA

cwBw



Linear Programming

The simplex algorithm is an exponential-time algorithm but often solves general linear programs quickly in practice. The first polynomial-time algorithm was the ellipsoid

algorithm, which runs slowly in practice.

A second class of polynomial-time algorithms are known as interior-point methods.

For large inputs, the performance of interior-point algorithms can be competitive with, and sometimes faster than, the simplex algorithm.

Linear programming (LP) is in P

6



An Example of Linear Programming

Maximize

x1 + x2 (29.11)

subject to

4x1 – x2 8 (29.12)

2x1 + x2 10 (29.13)

5x1 – 2x2 -2 (29.14)

x1, x2 0. (29.15)

7Chang’s illustration

constraints

Objective Function



LP Solvers

Web interface

http://www.zweigmedia.com/RealWorld/simplex.html

GNU Linear Programming Toolkit

A set of routines written in C and organized in the

form of a callable library

LP Solve

Library in C or R.

CBC (Coin-OR Branch and Cut)

A set of routines written in C++ and can be used as a

callable library or a stand-alone executable.

Matlab8



Linear Programming (LP) Relaxation

One direction for solving NP-hard problems.

Formulate the NP-hard problem into integer

programming.

Relax each integral constraints into linear constraints.

That is, reformulate the integer programming into linear

programming.

Solve the linear programming in polynomial time and

obtain linear solutions.

Reconstruct the integer solution using those linear

solutions.

9



A Typical Flow of LP-relaxation

The critical point is usually the rounding stage.

10

Integer

Programming

Integral solution

Linear Programming

Fractional solution

LP-Solver

Rounding

Stage



Vertex Cover Problem

The vertex cover problem is to find a

vertex cover of minimum size in a given

undirected graph.

A subset of vertices covering all edges.

11



Weighted Vertex Cover Problem

The weighted vertex cover problem asks for a

vertex cover of minimum total weight in an

undirected graph.

Each vertex is associated with a weight.

12

5 4

10

1 2

95



Integer Programming for Weighted Vertex

Cover

Suppose that we associate a variable x(v) with each vertex

v V.

x(v) {0, 1} for each v V, where x(v) = 1 if v being in the

vertex cover and x(v) = 0 otherwise.

For any edge (u, v), at least one of u and v must be in the

vertex cover. Thus, x(u) + x(v) 1.

(35.14)each for }1,0{)(

(35.13)),(each for 1)()(

subject to

)12.35()()( Minimize

Vvvx

Evuvxux

vxvwVv

13



Linear-Programming Relaxation

Suppose that we remove the constraint that x(v) {0, 1} and

replace it with 0 x(v) 1.

We then obtain the following linear program, which is

known as linear-programming (LP) relaxation.

)18.35(.each for 0)(

)17.35(each for 1)(

)16.35(),(each for 1)()(

subject to

)15.35()()( Minimize

Vvvx

Vvvx

Evuvxux

vxvwVv

14



A Lower Bound

Any feasible solution to the 0-1 integer program

in lines (35.12)-(35.14) is also a feasible solution

to the linear program in lines (35.15)-(35.18).

Therefore, the optimal solution of the linear

program is a lower bound on the optimal

solution of the integer program.

That is, a lower bound on the weight of an optimal

solution to the minimum-weight vertex-cover problem.

15



A 2-Approximation Rounding Algorithm

16

The following algorithm uses the optimal solution of the

linear program to construct an approximation solution to

the minimum-weight vertex-cover problem:

四捨五入法



Feasibility

17

Does this algorithm always output a feasible

solution?

That is, are all edges covered by at least one vertex?



Feasibility

To verify that C is a vertex cover, consider any

edge (u, v) E.

W know that x(u) + x(v) 1, which implies that at

least one of is at least ½ .

Therefore, at least one of u and v will be included in

the vertex cover, and so every edge will be covered.

)( and )( vxux

18



Approximation Algorithm

Criterion 1: feasibility

Yes.

Criterion 2: tractability

Yes. LP is solvable in polynomial time.

Criterion 3: quality -- ??

The solution’s quality is provably not too far from

that of an optimal solution.

19



Quality

Let C* be an optimal solution to the minimum-weight

vertex-cover problem.

Let z* be the optimal solution of the linear program.

Since an optimal vertex cover is a feasible solution to the

linear program, z* must be a lower bound on w(C*), that is

z* C*.

Let C be the approximation solution.

Since C* C, z* C* C.

20

Vv

vxvwz )()(*



Proof (cont.)

*

2/1)(:

2/1)(:

2

)()(2

)()(2

)()(2

)(

)(

z

vxvw

vxvw

vxvw

vw

vwC

Vv

Vv

vxVv

vxVv

Cv

21



Proof (cont.)

*

2/1)(:

2/1)(:

2

)()(2

)()(2

)()(2

)(

)(

z

vxvw

vxvw

vxvw

vw

vwC

Vv

Vv

vxVv

vxVv

Cv

22

since each 2x(v) at least 2*1/2



Proof (cont.)

*

2/1)(:

2/1)(:

2

)()(2

)()(2

)()(2

)(

)(

z

vxvw

vxvw

vxvw

vw

vwC

Vv

Vv

vxVv

vxVv

Cv

23



Proof (cont.)

algorithm.ion approximat-2 a is This

hence and ,2*/

have we,* *z Since

2

)()(2

)()(2

)()(2

)(

)(

*

2/1)(:

2/1)(:

CC

CC

z

vxvw

vxvw

vxvw

vw

vwC

Vv

Vv

vxVv

vxVv

Cv

24



LP-relaxation

The critical point is usually the rounding stage.

25

Integer

Programming

Integral solution

Linear Programming

Fractional solution

LP-Solver

Rounding

Stage



MaxSAT

Given an SAT instance, find a truth assignment to satisfy as much clauses as possible. 弟弟妹妹演算法：Randomly assign true or false to each

variable with probability 1/2.

We have shown that 弟弟妹妹演算法 is a 0.5-approximation algorithm.

Next, we will introduce a 哥哥姊姊演算法, which guarantees a better 0.637-approximation.

)()()( 43143221 xxxxxxxx

26



Integer Programming

Let’s try formulate the MaxSAT problem into an

integer programming.

)()()( 43143221 xxxxxxxx

27

.0

;1

.0

;var1

otherwise

satisfiedisCclausey

and

otherwise

trueisxiablex

Let

j

i

i

i

　　

　　　　　　

　　

　　　　　　　



Integer Programming

Let’s try formulate the MaxSAT problem into an

integer programming.

28 Relax the integral constraints.

)()()( 43143221 xxxxxxxx

;11,0

;11,0

;1)1(..

...max 21

mjy

njx

mjyxxts

yyy

i

i

iiCiiCi

m

jj

　　　　　　　　　　　　　　

　　　　　　　　　　　　　　

　　　　

　　　



Relaxation: Linear Programming

Solve the following linear programming using

existing algorithm.

29



LP-relaxation

How to infer the integral solutions from linear

solutions?

30

Integer

Programming

Integral solution

Linear Programming

Fractional solution

LP-Solver

Rounding

Stage



Randomized Rounding

1. Obtain the optimal linear solutions 𝒙𝟏∗ , 𝒙𝟐

∗ , … , 𝒙𝒏∗

2. for i = 1 to n

set xi to be true with probability 𝒙𝒊∗

Feasibility: Yes.

Tractability: Yes. Polynomial time.

Quality: ??

31



Analysis

We will show that

32

*

*

11satisfied is Pr

,

111satisfied is Pr

jj

C

j

j

ye

C

Therefore

yC

Cj

j

eCCE

yyC

yyeCE

m

m

/11/][

...

.../11][

*

**

1

*

**

1



Analysis

33

||

**

**

*

||

1

)1(dunsatisfie is Pr

111satisfied is Pr

j

jj

j

j

C

j

ijciijci

ci

i

ci

ij

C

j

j

C

xx

xxC

proof

yC

C



Analysis

34

k

aaaaaa kk

k

21/1

21

||**

||**

**

||

11

||

)1(

)1(dunsatisfie is Pr

j

jj

j

jj

jj

C

j

ci ici i

C

j

ci ici i

ci

i

ci

ij

C

xx

C

xx

xxC



Linear Programming

The summation is bounded by the constraint in

linear programming.

35



Analysis

36

||C

j

*

i

||C

j

ci ci

*

i

*

i

||C

j

ci

*

ici

*

i

ci

*

i

ci

*

ij

j

j

j j

j

jj

jj

||C

y

||C

xx

||C

x)x(

x)x(sfied is unsatiC

1

)1(1

1

1Pr



Analysis

37

*

*

111

11satisfied is Pr

j

j

j

j

yC

C

yC

C

j

C

j

j



Simulation

38

f (r )

f (1) ¢r

r = 1r = 0

.)1

1(1)(

have we,integer positiveany and 10any for Then

.)1(1)(Let

rk

rf

kr

k

rrf k



Lemma

39

f (r )

f (1) ¢r

r = 1r = 0



A Better Approximation

Algorithm

40

*

*

*

11

111

11satisfied is Pr

j

j

j

j

j

ye

yC

C

yC

C

j

C

j

j

e

r

r

r

1lim



Analysis

We have shown that

41

*11satisfied is Pr jj y

eC

...673.0/11/][

...

.../11][

*

**

1

*

**

1

eCCE

yyC

yyeCE

m

m



Randomized Rounding Randomized rounding implicitly performs local

search around the linear optimal solution

42

https://www.google.com.tw/imgres?imgurl=http://glassproperties.com/optimization/non-

convex_optimization.gif&imgrefurl=http://www.snipview.com/q/Convex%2520optimization&h=384&w=512&tbni

d=kAYmrhtx8IpcUM:&docid=7wx__AWoEDWEbM&hl=zh-

TW&ei=HuhoVvbnOsO8mgXL3K2QCQ&tbm=isch&ved=0ahUKEwj27on_o9DJAhVDnqYKHUtuC5IQMwgiKA

cwBw



Summary

The MaxSAT problem can be solved by a

randomized (1-1/e)-approximation algorithm.

1-1/e = 0.637… > 0.5

哥哥姊姊演算法 is better than the 0.5-approximation

弟弟妹妹演算法 in theory and in practice.

Keep in mind all randomized approximation

algorithms have to be repeated.

43



Observation

In 弟弟妹妹演算法 , each clause is satisfied with

probability.

If the number of clauses ≥ 2, it’s a 0.75

approximation algorithm.

In 哥哥姊姊演算法, each clause is satisfied with

probability

If the number of clauses ≤ 2, it’s a 0.75

approximation algorithm.44

*111satisfied is Pr

j

j

yC

C

C

j

j

jCjC

2

11satisfied is Pr



全家團結演算法

Pick the best solution between 弟弟妹妹演算法and 哥哥姊姊演算法.

全家團結演算法 is a 0.75-approximation

algorithm.

45



An Integer Programming for Set

Cover Let’s formulate the set cover problem into integer

programming.

Let xi be the i-th set.

Objective function?

Constraint function?

46

S1

S3S4

S6

S2

S5




Cover

Let’s formulate the set cover problem into integer

programming.

Let xi be the i-th set.

Let Si be the sets covering

the i-th element.

}.1,0{ where,1 Subject to

Minimize1

k

Sk

k

N

k

k

xx

x

i

47

S1

S3S4

S6

S2

S5




Cover

}.1,0{ where

...

,1

,1

,1

,1

Subject to

Minimize

421

51

41

31

6

1

k

k

k

x

xxx

xx

xx

xx

x

48

S1

S3S4

S6

S2

S5



An iterative LP-relaxation algorithm

]1,0[ ,1 Subject to

Minimize1

k

Sk

k

N

k

k

yy

y

i

49


Minimize1

k

Sk

k

N

k

k

xx

x

i

Relax the integral constraint.

Then??




Step 1. Formulate the integer programming.

Step 2. Relax the integer constraint to linear constraint.

50


Minimize1

k

Sk

k

N

k

k

xx

x

i

]1,0[ ,1 Subject to

Minimize1

k

Sk

k

N

k

k

yy

y

i






Step 3. Obtain the optimal solution of the LP problem.

y1 = 0.2, y2=0.7, y3=0.5, …

But we only care about the solution of xi.

51


Minimize1

k

Sk

k

N

k

k

xx

x

i

]1,0[ ,1 Subject to

Minimize1

k

Sk

k

N

k

k

yy

y

i






Step 3. Obtain the optimal solution of the LP problem and

perform randomized rounding.

Set xi = 1 with probability yi.

52








Set xi = 1 with probability yi.

Thus, we will have x1 = 0, x2, = 1, …

Any problem??

53


Minimize1

k

Sk

k

N

k

k

xx

x

i

]1,0[ ,1 Subject to

Minimize1

k

Sk

k

N

k

k

yy

y

i



Violation of Feasibility

Criterion 1: feasibility -- No

This may not be a feasible solution.

How to overcome this problem??

54



Monte Carlo Algorithm





Step 4. The randomized rounding may invalidate some integral

inequalities. We repeat Steps 1, 2, and 3 for those unsatisfied

inequalities until all of them are satisfied.

Otherwise.

iteration;any in 1 set to is if ,0

,1 Assign :Solution Final

k

k

k x

x

x

55



An iterative LP-relaxation

algorithm

56

Integer Linear

Programming

Integral solution

Linear Programming

Fractional solution

relaxation

LP-Solver

yes

noAll inequalities satisfied?

Ting ChenKui Zhang



Final Solution

Union of all selected sets in each iteration.

Otherwise.



k

k

k x

x

x

57

t 1 2 3 Final

x1 1 0 1 1

x2 0 1 1 1

x3 0 0 0 0

x4 0 0 0 0

x5 1 0 0 1



http://www.zweigmedia.com/RealWorld/simple

x.html

58 }.1,0{ where

...

,1

,1

,1

,1

Subject to

Minimize

421

51

41

31

6

1

k

k

k

x

xxx

xx

xx

xx

x

S1

S3S4

S6

S2

S5



Analysis

How many iterations needed?

How about the solution quality?

59

Integer Linear

Programming

Integral solution

Linear Programming

Fractional solution

LP-Solver

yes




Analysis

How many (expected) iterations needed?

What is the probability of repeating?

60

Integer Linear

Programming

Integral solution

Linear Programming

Fractional solution

LP-Solver

yes



& Information EngineeringAn Integer Programming for Set

Cover


When should we repeat?

Any one of the constraint is not satisfied.

}.1,0{ where

...

,1

,1

,1

,1

Subject to

x Minimize

421

51

41

31

6

1k

k

kx

xxx

xx

xx

xx

61

S1

S3S4

S6

S2

S5



62

Analysis of the Iterative LP-

Relaxation Algorithm (2/3)

r

k

ky1

)1(

The probability that one constraint (of r terms) is not satisfied

is

]1,0[ ,1 Subject to

Minimize1

k

Sk

k

N

k

k

yy

y

i

}.1,0{ where

...

,1

,1

,1

,1

Subject to

x Minimize

421

51

41

31

6

1k

k

kx

xxx

xx

xx

xx



63

Analysis of the Iterative LP-Relaxation

Algorithm (2/3)

The probability that any constraint is not satisfied is

r

xxx

rrrxxx

21

1

)( 21


Minimize1

k

Sk

k

N

k

k

xx

x

i

]1,0[ ,1 Subject to

Minimize1

k

Sk

k

N

k

k

yy

y

i

rrr

k

r

k

kk

rr

yy

111)1(

1 1



64


Algorithm (2/3)

.1

11)1( 1

1 1

e

rr

yy

rrr

k

r

k

kk

The probability that any constraint is not satisfied is

r

xxx

rrrxxx

21

1

)( 21

e

r

r

r

1lim


Minimize1

k

Sk

k

N

k

k

xx

x

i

]1,0[ ,1 Subject to

Minimize1

k

Sk

k

N

k

k

yy

y

i



Satisfying Probability v.s. Iterations The probability of any constraint satisfied in t iterations is

The probability that all K constraints are satisfied in t

iterations is

Otherwise.



k

k

k x

x

x

65

t 1 2 3 …

C1 X

C2 X X

C3 X X X X

Kt

e

11

t

e

11



66


Algorithm (2/3)

111lim

e

r

r

r

The probability that all K constraints are satisfied in t

iterations is

.1

11

11

1t

tt

e

Ke

K

e

t

K

t

Kt

eeee

After t = lnK iterations, the algorithm stops and returns a

solution with probability e-1.



Analysis How many iterations needed?

O(log K)


67

Integer Linear

Programming

Integral solution

Linear Programming

Fractional solution

LP-Solver

yes




Analysis of the Iterative LP-

Relaxation Algorithm (1/3)

]1,0[ ,1 Subject to

Minimize

),(

1

k

PPDk

k

N

k

k

yy

y

ji

).()(11

IPOPTxyLPOPTN

k

k

N

k

k

68


Minimize

),(

1

k

PPDk

k

N

k

k

xx

x

ji

Denote the optimal solution of the relaxed LP problem as

OPT(LP).

Since the solution space of LP includes that of IP,

OPT(LP)≦OPT(IP),

OPT(LP) severs as lower bound for the algorithm



69

Analysis of the Iterative LP-Relaxation Algorithm (3/3)

Let the objective values of OPT(IP) returned at each

iteration be Z1 , Z2 , …, Zt .

).()(ln

)(

EEE[C]11

IPOPTKO

LPOPTt

Zxt

p

p

N

k

k

).(EE1

,

1

, LPOPTyxZN

k

ki

N

k

kii

The expected size of set cover found by this algorithm is



Analysis


O(log K), wher K is the number of elements.


The iterative LP-relaxation algoirhtm is an O(log

K)-approximation algorithm.

70



Comparison with Greedy Algorithm

Iteratively selects a set S covering most elements.

71



Conclusion

Deterministic algorithm runs very fast but does

not make use of the speed of modern CPU.

The randomized algorithm with any sort of

intelligence (e.g., LP relaxation) often outputs

better solutions by making good use of CPU.

72



Conclusion

LP-relaxation is one useful technique for

solving NP-hard problems.

It is often used when you feel the linear solutions

are helpful to infer the integral solutions.

Although the entire process is a bit complicated,

the rounding algorithm is the key.

73



Appendix

74



75

設 f(x)與g(x)都是可微分函數,c是常數

(1)若 ,k為定實數,則

(2)若 ,n為正整數,則

(3)

(4)

nkxf )(

kxf )( 0)( xfdx

d

1)( nnxxfdx

d

)()]([ xfdx

dcxcf

dx

d

)()()]()([ xgdx

dxf

dx

dxgxf

dx

d

Proven by Differentiation

Source: web.cc.ntnu.edu.tw/~49340133/PPT/P1021_1332_2398.ppt



)].()[()()]([)]()([ xgdx

dxfxgxf

dx

dxgxf

dx

d

76

(7)連鎖規則若 ,令 ,則

.0)(,)(

)]([)()()]([

))(

)((

2

xgxg

xgdx

dxfxgxf

dx

d

xg

xf

dx

d

))(()( xgfxF )(xgu

udx

duf

du

dxgf

dx

dxF

dx

d)())(()(

(5)

(6)

Quick Review of Differentiation



2x

77

一個函數圖形凹向，一般可分為：1.凹口向上 2.凹口向下兩種．

x

y

0

PQ

f(x)=

P

y

x0

Q

f(x)=


2x



78

設函數f(x)在內a＜x＜b的每一點的二階導數都存在,

(1) 對於a＜x＜b內的每一點, f”(x)＞0 ,則

f(x)在a＜x＜b內凹口向上(向上凹).

(2) 對於a＜x＜b 內的每一點, f”(x)＜0,則

f(x)在a＜x＜b內凹口向下(向下凹).




79



80

Linear Programming Relaxation

Documents

Transcript of Linear Programming Relaxation