Sparse Linear Algebra: LU Factorization - McMaster …cs777/presentations/LU...Once the L and U...

Sparse Linear Algebra: LU Factorization

Kristin Davies

Peter He

Feng Xie

Hamid Ghaffari

April 4, 2007

Outline

� Introduction to LU Factorization (Kristin)

� LU Transformation Algorithms (Kristin)

� LU and Sparsity (Peter)

� Simplex Method (Feng)

� LU Update (Hamid)

Introduction – Transformations – Sparsity – Simplex – Implementation

Introduction – What is LU Factorization?

� Matrix decomposition into the product of a

lower and upper triangular matrix:

LUA =

� Example for a 3x3 matrix:

=

33

2322

131211

333231

2221

11

333231

232221

131211

00

00

00

u

uu

uuu

lll

ll

l

aaa

aaa

aaa


Introduction – LU Existence

� LU factorization can be completed on an

invertible matrix if and only if all its principle

minors are non-zero

� Recall: invertible Recall: principle minors( ) ( ) ( )

333231

232221

131211

3331

1311

3332

2322

2221

1211

332211

det

detdetdet

detdetdet

aaa

aaa

aaa

aa

aa

aa

aa

aa

aa

aaa

=

333231

232221

131211

aaa

aaa

aaa

A

IBAAB ==

s.t.existsBifinvertibleisA


Introduction – LU Unique Existence

� Imposing the requirement that the diagonal of

either L or U must consist of ones results in

unique LU factorization

=

100

10

1

0

00

00

0

1

01

001

23

1312

333231

2221

11

33

2322

131211

3231

21

333231

232221

131211

u

uu

lll

ll

l

u

uu

uuu

ll

l

aaa

aaa

aaa


Introduction – Why LU Factorization?

� LU factorization is useful in numerical analysis

for:

– Solving systems of linear equations (AX=B)

– Computing the inverse of a matrix

� LU factorization is advantageous when there is

a need to solve a set of equations for many

different values of B


Transformation Algorithms

� Modified form of Gaussian elimination

� Doolittle factorization – L has 1’s on its diagonal

� Crout factorization – U has 1’s on its diagonal

� Cholesky factorization – U=LT or L=UT

� Solution to AX=B is found as follows:

– Construct the matrices L and U (if possible)

– Solve LY=B for Y using forward substitution

– Solve UX=Y for X using back substitution


Transformations – Doolittle

� Doolittle factorization – L has 1’s on its diagonal

� General algorithm – determine rows of U from top to bottom; determine columns of L from left to right

for i=1:n

for j=i:n

LikUkj=Aij gives row i of U

end

for j=i+1:n

LjkUki=Aji gives column i of L

end

end


Transformations – Doolittle example

( ) 22

0

0001

824

364

212

00

0

1

01

001

11

11

33

2322

131211

3231

21

=⇒=

−−

−

−−

=

u

u

u

uu

uuu

ll

l

for j=i:n

LikUkj=A

ijgives row i of U

end

2,1

824

364

212

00

0

1

01

001

1312

33

2322

131211

3231

21

−=−=⇒

−−

−

−−

=

uu

u

uu

uuu

ll

l

Similarly



( ) 2424

0

0

2

01

824

364

212

00

0

212

1

01

001

212121

33

2322

3231

21

−=⇒−=⇒−=

−−

−

−−

=

−−

lll

u

uu

ll

l

for j=i+1:n

LjkUki=A

jigives column i of L

end

2

824

364

212

00

0

212

1

01

001

31

33

2322

3231

21

−=⇒

−−

−

−−

=

−−

l

u

uu

ll

l

Similarly



−−

−

−−

=

−−

824

364

212

00

0

212

1

01

001

33

2322

3231

21

u

uu

ll

l

−−

−

−−

=

824

364

212

00

0

1

01

001

33

2322

131211

3231

21

u

uu

uuu

ll

l

−−

−

−−

=

−−

−

−

824

364

212

00

0

212

12

012

001

33

2322

32 u

uu

l

−−

−

−−

=

−

−−

−

−

824

364

212

00

140

212

12

012

001

3332 ul

−−

−

−−

=

−

−−

−−

−

824

364

212

00

140

212

112

012

001

33u

−−

−

−−

=

−

−−

−−

−

824

364

212

300

140

212

112

012

001

1st row of U

2nd row of U

1st coln of L2nd coln of L

3rd row of U



� Execute algorithm for our example (n=3)

for i=1:n

for j=i:n

LikUkj=A

ijgives row i of U

end

for j=i+1:n

LjkUki=A


end

end

i=1

j=1 �� L1kUk1=A

11gives row 1 of U

j=2 �� L1kUk2=A

12gives row 1 of U

j=3 �� L1kUk3=A

13gives row 1 of U

j=1+1=2 �� L2kUk1=A

21gives column 1 of L

j=3 �� L3kUk1=A


i=2

j=2 �� L2kUk2=A

22gives row 2 of U

j=3 �� L2kUk3=A

23gives row 2 of U

j=2+1=3 �� L3kUk2=A


i=3

j=3 �� L3kUk3=A

33gives row 3 of U


Transformations – Crout

� Crout factorization – U has 1’s on its diagonal

� General algorithm – determine columns of L from left to right; determine rows of U from top to bottom (same!?)

for i=1:n

for j=i:n

LjkUki=Aji gives column i of L

end

for j=i+1:n

LikUkj=Aij gives row i of U

end

end


Transformations – Crout example

( ) 22

0

0

1

00

824

364

212

100

10

1

0

00

1111

23

1312

333231

2221

11

=⇒=

−−

−

−−

=

ll

u

uu

lll

ll

l for j=i:n

LjkUki=A


end

Similarly

4,4

824

364

212

100

10

1

0

00

3121

23

1312

333231

2221

11

−=−=⇒

−−

−

−−

=

ll

u

uu

lll

ll

l


Transformations – Solution

� Once the L and U matrices have been found, we can easily solve our system

AX=B �� AUX=B

=

3

2

1

3

2

1

333231

2221

11

0

00

b

b

b

y

y

y

lll

ll

l

=

3

2

1

3

2

1

33

2322

131211

00

0

y

y

y

x

x

x

u

uu

uuu

nifora

xlb

y

l

by

ii

i

j

jiji

i ,...2

1

1

11

11

=

−

=

=

∑−

=1),...1(

1 −=

−

=

=

∑+=

niforu

yuy

x

u

yx

ii

n

ij

jiji

i

nn

nn

LY=B UX=Y

Forward Substitution Backward Substitution


References - Intro & Transformations

� Module for Forward and Backward Substitution. http://math.fullerton.edu/mathews/n2003/BackSubstitutionMod.hml

� Forward and Backward Substitution. www.ac3.edu/papers/Khoury-thesis/node13.html

� LU Decomposition. http://en.wikipedia.org/wiki/LU_decomposition

� Crout Matrix Decomposition. http://en.wikipedia.org/wiki/Crout_matrix_decomposition

� Doolittle Decomposition of a Matrix. www.engr.colostate.edu/~thompson/hPage/CourseMat/Tutorials/C

ompMethods/doolittle.pdf


Definition and Storage of Sparse Matrix

� sparse … many elements are zero

for example: diag(d1,…,dn), as n>>0.

� dense … few elements are zero

� In order to reduce memory burden, introduce

some kinds of matrix storage format



� Regular storage structure.

row 1 1 2 2 4 4

list = column 3 5 3 4 2 3

value 3 4 5 7 2 6

– A standard sparse matrix format

Subscripts:1 2 3 4 5 6 7 8 9 10 11

Colptr: 1 4 6 8 10 12

Rowind: 1 3 5 1 4 2 5 1 4 2 5

Value: 1 2 5 -3 4 -2 –5-1 –4 3 6



1 -3 0 -1 0

0 0 -2 0 3

2 0 0 0 0

0 4 0 –4 0

5 0 –5 0 6


Structure Decomposition of Sparse Matrix (square)

�

� Matrix is of permutation and

A G(A) Adj(G)↔ ↔

iAdj(G) {0,1}

j

∈

TG(A) G(P AP)=P

T T

T T

opt{P AP P:G(P AP) G(A)}

opt{P adj(G)P P :G(P adj(G)P) G(A)}

= ⇔

=



� 1. There is no loop in G(A)

Step 1. No loop and t = 1

Step 2. output vertices

Step 3.

∃

{ }t

t t t V(G)nt 1 2S i i . . . i ⊂=

t

t

t t tn1 2

adj(G) adj(G) \ adj(G)t t t

n1 2

i i . . . i

i i . . . i

=

t=t+1, returning to step 2.

� After p times,

(separation of set).

such that is a lower triangular

block matrix . Therefore is a lower



{ } p1 21,2,...,n S S ... S= ∪ ∪ ∪

p1 2

TT T Ts s sP P ,P ,...,P

∃ =

TP adj(G)P

TP AP

triangular block matrix .

� 2.There is a loop in G(A).

If the graph is not strongly connected, in the

reachable matrix of adj(A), there are naught

entries.

Step 1. Choose the j-th column, t = 1,and



∩

= = ≤ ≤ =T(R R )ti

S (j) i 1,1 i nj



is closed on strong connection.

Step 2: Choose the -th column ( ),

, t=t+1, returning step 1.

� After p times,

{ }t

t t tn1 2

t

i i . . . i

( j S 1 2 . . . n ,

=

∈ ⊂

tS )

1j 1j j≠

1j j=

{ } p1 21,2,...,n S S ... S= ∪ ∪ ∪

is a lower triangular block matrix. Therefore

is a lower triangular block matrix.

� Note of case 2 mentioned above.



p1 2

T T T T Ts s sP (P P . . . P ) , P adj(G)P∃ = ∋

TP AP

= =

11 12 1p

21 22 2pT

p1 pp

ˆ ˆ Â A . . . A

ˆ ˆ Â A . . . AA P adj(G)P

. . . . . .

ˆ Â . . . . A

.

p pij

ij ij

ij

B (b ) R ,

0 i j

ˆb 1 A 0.

ˆ0 A 0

×

= ∈

= =

= = ≠ ⇒

= =



There is no loop of G(B). According to similar

analysis to case 1,

. Therefore, is a lower triangular block

matrix.



p pTp p1 2 1 2

.P(j j . . . j ) BP(j j . . . j ) R

.

.

×

××

= ∈× ×× × ×

TP AP

� Reducing computation burden on solution to

large scale linear system

� Generally speaking, for large scale system,

system stability can easily be judged.

� The method is used in hierarchical optimization

of Large Scale Dynamic System



Dulmage-Mendelsohn Decomposition

� such that


Structure Decomposition of Sparse Matrix (General)

P,Q∃

T

h

s

v

A

P AQ A X

A

x x

=

� Further, fine decomposition is needed.

the block diagonal form

the block diagonal form

the block upper triangular form.

� D-M decomposition can be seen in reference

[3].

� Computation and storage: Minimum of filling in

The detail can be see in reference [2].


Structure Decomposition of Sparse Matrix (General)

h

v

s

A

A

A

→

→→

� left-looking. kth stage computes kth column of L and U

1. L = I

2. U = I

3. for k = 1:n

4. s = L \ A(:,k)

5. (partial pivoting on s)

6. U(1:k,k) = s(1:k)

7. L(k:n,k) = s(k:n) / U(k,k)

8. end


LU Factorization Method :Gilbert/Peierls

� THEOREM(Gilbert/Peierls). The entire

algorithm for LU factorization of A

with partial pivoting can be

implemented to run in O(flops (LU)+ m)

time on a RAM

where m is number of the nonzero

entries of A.

� Note: The theorem expresses that the LU

factorization will run in the time

within a constant factor of the best

possible, but it does not say what the


LU Factorization Method :Gilbert/Peierls

x = b;

for j = 1:n

if (x(j) 0) , x(j+1:n) =

x(j+1:n) - L(j+1:n,j) * x(j);

end; ---Total time: O(n+flops)


Sparse lower triangular solve, x=L\b

≠



�

�

� Let have an edge

if

� Let and

� Then

---Total time: O(flops)

i ib 0 x 0≠ ⇒ ≠

j ij ix 0 l 0 x 0≠ ∧ ≠ ⇒ ≠

G(L) j i→ijl 0≠

{ }ii b 0β = ≠ { }i

i x 0χ = ≠

G(L)Reach ( )χ = β

� J. R. Gilbert and T. Peierls, Sparse partial pivoting in time proportional to arithmetic operations, SIAM J.

Sci. Statist. Comput., 9 (1988), pp. 862-874

� Mihalis Yannakakis’ website:

http://www1.cs.columbia.edu/~mihalis

� A. Pothen and C. Fan, Computing the block triangular

form of a sparse matrix, ACM Trans. On Math. Soft.

Vol. 18, No.4, Dec. 1990, pp. 303-324


References

� Problem size determined by A

� On the average, 5~10 nonzeros per column


Simplex Method

– Problem size

{ }min , 0,T m nc x Ax b x A R

×= ≥ ∈

greenbea.mps

of NETLIB


Simplex Method

– Computational Form, Basis

Note: A is of full row rank and m<n

Basis (of Rm) : m linearly independent columns of A

Basic variables

{ }min , 0,T m nc x Ax b x A R

×= ≥ ∈

0 2

1 1

1 1

0 2

x1 x2 x3 x4

β index set of basic variables

γ index set of non-basic variables

B:=Aβ basis

R:=Aγ non-basis (columns)

[ | ], ,x c

A B R x cx c

β β

γ γ

= = =


Simplex Method

– Notations

Basic feasible solution

If a problem has an optimal solution, then there is a basic

solution which is also optimal.

Ax b=

1( )x B b Rxβ γ−= −

, 0,0 x xx A bγ ≥ ==

Bx Rx bβ γ+ =


Simplex Method

– Basic Feasible Solution

Objective value :

Optimality condition :

( )1 1

0

constant Reduced Costs

T T T

T T T

T

c x c x c x

c c R x

c

B b c B

d x

β β γ γ

β γ γ

γ

β− −

= +

= −

= +

+��

for all0j

d j γ≥ ∈


Simplex Method

– Checking Optimality

non-basic variables

� Choose xq (incoming variable) s.t. dq < 0

� increase xq as much as possibleObjective value

0

Tc d xγ+

1 0

0

m

q

n

x

x

x

+

=

↑

=

⋮

⋮

basic variables

1

p

n

x

x

x

↑

→

↑

⋮

⋮

objective value is unbounded

basic variables remain feasible

(≥ 0) even if xq �� ∞


Simplex Method

– Improving Basic Feasible Solution

non-basic variables

� Choose xq (incoming variable) s.t. dq < 0

� increase xq as much as possible

Objective value

0

Tc d xγ+

1 0

0

m

q

n

x

x

x

+

=

↑

=

⋮

⋮

basic variables

1

p

n

x

x

x

↑

→

↑

⋮

⋮

basic variables

1

p

n

x

x

x

↑

↓

↑

⋮

⋮

goes to 0 first

(outgoing variable)

Unbounded solution neighboring improving basis


Simplex Method

– Improving Basic Feasible Solution

1,..., ,...,p m

B b bb =

[ ]1,..., ,...,m

aB b b=

Neighboring bases :


Simplex Method

– Basis Updating

1,..., ,...,p m

B b bb =

[ ]1,..., ,...,m

aB b b=1

Writem

i

i

i

v b Bva=

= =∑


Simplex Method

– Basis Updating

1( )v B a

−=

(as the linear combination of the bases)

1,..., ,...,p m

B b bb =

[ ]1,..., ,...,m

aB b b=1

Writem

i

i

i

v b Bva=

= =∑


Simplex Method

– Basis Updating

1( )v B a

−=

1 1 1

1( 0

1, where , , , , ,

)

,

Tp p m

p

ip

i

p p

p pi p

p

p p p

v v v vb B

v v v v

vb v

v v

v

b a

η η−

≠

+ = = −

= − ≠

− − −

∑

⋯ ⋯

1,..., ,...,p m

B b bb =

[ ]1,..., ,...,m

aB b b=1

Writem

i

i

i

v b Bva=

= =∑


Simplex Method

– Basis Updating

1( )v B a

−=

1 1 1

1

1, ,where , , , , ,

p

i

ip pi p

Tp p

p

m

p p p p p

vb

v v

v v v vB

v v v v v

b a

b η η

≠

− +

= −

= = − − − −

∑

⋯ ⋯

vp : pivot element

1,..., ,...,p m

B b bb =

[ ]1,..., ,...,m

aB b b=1

Writem

i

i

i

v b Bva=

= =∑

1 1 1,..., , , ,., where ..,p p m

B BE E e e e eη− + = =


Simplex Method

– Basis Updating

(elementary transformation matrix)

1( )v B a

−=

1 1 1

1

1, ,where , , , , ,

p

i

ip pi p

Tp p

p

m

p p p p p

vb

v v

v v v vB

v v v v v

b a

b η η

≠

− +

= −

= = − − − −

∑

⋯ ⋯

1,..., ,...,p m

B b bb =

[ ]1,..., ,...,m

aB b b=1

Writem

i

i

i

v b Bva=

= =∑

1 1 1,..., , , ,., where ..,p p m

B BE E e e e eη− + = =

1 1B EB

− −=


Simplex Method

– Basis Updating

1( )v B a

−=

1 1 1

1

1, ,where , , , , ,

p

i

ip pi p

Tp p

p

m

p p p p p

vb

v v

v v v vB

v v v v v

b a

b η η

≠

− +

= −

= = − − − −

∑

⋯ ⋯

1 1 1

1

1

, ..., , , , ...,

1

1

1 0 1 01

11

0 1 10 1

p p m

p

m

p

m

E e e e eη

η

η

η

η

η

η

− + =

=

=

⋱

⋱

⋱ ⋮ ⋱ ⋮⋱ ⋮

⋯

⋮ ⋱ ⋮ ⋱⋮ ⋱


Simplex Method

– Basis Updating

ETM

(Elementary Transformation Matrix)

Basis tends to get denser after each update

1 1 1 11

1

p

p p p p

m m m p m

w w w

Ew w w

w w w

η η

η η

η η

+ = = +

⋱ ⋮ ⋮

⋱ ⋮ ⋮

1 1( )B EB− −=

Ew = w if wp = 0


Simplex Method

– Basis Updating

Steps Major ops

1. Find an initial feasible basis B

2. Initialization

3. Check optimality

4. Choose incoming variable xq

5. Choose outgoing variable xp

6. Update basis

1B b

−

1Tc Bβ−

1

qB a−

1EB

−


Simplex Method

– Algorithm

Steps Major ops

1. Find an initial feasible basis B

2. Initialization

3. Check optimality

4. Choose incoming variable xq

5. Choose outgoing variable xp (pivot step)

6. Update basis

1B b

−

1Tc Bβ−

1

qB a−

1EB

−


Simplex Method

– Algorithm

Choice of pivot (numerical considerations)

� resulting less fill-ins

� large pivot element

Conflicting goals sometimes

In practice, compromise.


Simplex Method

– Algorithm

Typical operations :

Challenge: sparsity of B-1 could be destroyed by basis update

Need a proper way to represent B-1

Two ways:

� Product form of the inverse (obsolete)

� LU factorization

1 1, TB w w B

− −


Simplex Method

– Typical Operations in Simplex Method

1

1 1( )k kB E E E−−= ⋯

� Reduce complexity using LU update

Side effect : more LU factors

� Refactorization

(reinstate efficiency and numerical accuracy)


Simplex Method

– LU Factorization

1 1( , )B BE B EB− −= =

Sparse LU Updates in Simplex Method

Hamid R. Ghaffari

April 10, 2007

Outline

LU Update MethodsPreliminariesBartels-Golub LU UPDATESparse Bartels-Golub MethodReid’s MethodThe Forrest-Tomlin MethodSuhl-Suhl MethodMore Details on the Topic

Revised Simplex Algorithm

Simplex Method Revised Simplex Method

Determine the current basis, d d = B−1b

Choose xq to enter the basis based onthe greatest cost contribution

c = c ′N − c ′BB−1N,{q|cq = min

t(ct)}

If xq cannot decrease the cost,d is optimal solution

cq ≥ 0, d is optimal solution

Determine xp that leaves the basis(become zero) as xq increases.

w = B−1Aq,{p di

wi= min

t

(dtwt

), wt > 0

}If xq can increase without causinganother variable to leave the basis,the solution is unbounded

If wp ≤ 0 for all i , the solution isunbounded.

Update dictionary. Update B−1

Note: In general we do not compute the inverse.

Problems with Revised Simplex Algorithm

I The physical limitations of a computer can become a factor.

I Round-off error and significant digit loss are common problems inmatrix manipulations (ill-conditioned matrices).

I It also becomes a task in numerical stability.

I It take m2(m− 1) multiplications and m(m− 1) additions, a total ofm3 −m floating-point (real number) calculations.

Many variants of the Revised Simplex Method have been designed toreduce this O(m3)-time algorithm as well as improve its accuracy.

Introducing Spike

I If Aq is the entering column, B the original basis and B the newbasis, then we have

B = B + (Aq − Bep)eTq ,

I Having LU decomposition B = LU we have

L−1B = U + (L−lAq − Uep)eTq ,

II How to deal with this?

The various implementations and variations of the Bartels-Golubgenerally diverge with the next step: reduction of the spiked uppertriangular matrix back to an upper-triangular matrix. (Chvatal, p150)

Bartels-Golub MethodIllustration

The first variant of the Revised Simplex Method was the Bartels-GolubMethod.

Bartels-Golub MethodAlgorithm

Revised Simplex Method Bartels-Golub

d = B−1b d = U−1L−1b

c = c ′N − c ′BB−1N,{q|cq = min

t(ct)}

c = c ′N − c ′BU−1L−1N,{q|cq = min

t(ct)}

cq ≥ 0, d is optimal solution cq ≥ 0, d is optimal solution

w = B−1Aq,{p di

wi= min

t

(dtwt

), wt > 0

} w = U−1L−1Aq,{p di

wi= min

t

(dtwt

), wt > 0

}If wp ≤ 0 for all i , the solution isunbounded.


Update B−1 Update U−1 and L−1

Bartels-Golub MethodCharacteristics

I It significantly improved numerical accuracy.

I Can we do better?

In sparse case, yes.

Bartels-Golub MethodCharacteristics

I It significantly improved numerical accuracy.

I Can we do better? In sparse case, yes.

Sparse Bartels-Golub Methodeta matrices

First take a look at the following facts:

Column-Eta factorization of triangular matrices:

1l21 1l31 l32 1l41 l42 l43 1

=

11

1l43 1

·

11l32 1l42 1

·

1l21 1l31 1l41 1

Single-Entry-Eta Decomposition:

1l21 1l31 1l41 1

=

1l21 1

11

·

11

l31 11

·

11

1l41 1

So L can be expressed as the multiplication of single-entry eta matrices,and hence, L−1 is also is the product of the same matrices withoff-diagonal entries negated.

Sparse Bartels-Golub MethodAlgorithm

Bartels-Golub Method Sparse Bartels-Golub Method

d = U−1L−1b d = U−1 ∏t ηtb

c = c ′N − c ′BU−1L−1N,{q|cq = min

t(ct)}

c = c ′N − c ′BU−1 ∏t ηtN,

{q|cq = mint

(ct)}cq ≥ 0, d is optimal solution cq ≥ 0, d is optimal solution

w = U−1L−1Aq,{p di

wi= min

t

(dtwt

), wt > 0

} w = U−1 ∏t ηtAq,{

p diwi

= mint

(dtwt

), wt > 0



Update U−1 and L−1 Update U−1 and create anynecessary eta matrices.If there are too many eta matrices,completely refactor the basis.

Sparse Bartels-Golub Method Advantages

I It is no more complex than the Bartels-Golub Method.

I Instead of just L and U, the factors become the lower-triangular etamatrices and U.

I The eta matrices were reduced to single-entry eta matrices.

I Instead of having to store the entire matrix, it is only necessary tostore the location and value of the off-diagonal element for eachmatrix.

I Refactorizations occur less than once every m times, so thecomplexity improves significantly to O(m2).

Sparse Bartels-Golub Method Disadvantages

I Eventually, the number of eta matrices will become so large that itbecomes cheaper to decompose the basis.

I Such a refactorization may occur prematurely in an attempt topromote stability if noticeable round-off errors begin to occur.

I In practice, in solving large sparse problem, the basis is refactorizedquite frequently, often after every twenty iterations of so. (Chvatal,p. 111)

I If the spike always occurs in the first column and extends to thebottom row, the Sparse Bartels-Golub Method becomes worse thanthe Bartels-Golub Method.

I The upper-triangular matrix will always be fully-decomposedresulting in huge amounts of fill-in;

I Large numbers of eta matrices;I O(n3)-cost decomposition;







I Large numbers of eta matrices;

I O(n3)-cost decomposition;







I Large numbers of eta matrices;I O(n3)-cost decomposition;

Ried Suggestion on Sparse Bartels-Golub Method

Rather than completely refactoring the basis, applying LU-decompositiononly to the part of that remained upper-Hessenberg.

Reid’s Method

Task: The task is to find a way to reduce that bump before attempting todecompose it;

Row singleton: any row of the bump that only has one non-zeroentry.

Column singleton: any column of the bump that only has onenon-zero entry.

Method:I When a column singleton is found, in a bump, it is moved to the top

left corner of the bump.

I When a row singleton is found, in a bump, it is moved to the bottomright corner of the bump.

Reid’s MethodColumn Rotation

Reid’s MethodRow Rotation

Reid’s MethodCharacteristics

Advantages:

I It significantly reduces the growth of the number of eta matrices inthe Sparse Bartels-Golub Method

I So, the basis should not need to be decomposed nearly as often.

I The use of LU-decomposition on any remaining bump still allowssome attempt to maintain stability.

Disadvantages:

I The rotations make absolutely no allowance for stability whatsoever,

I So, Reid’s Method remains numerically less stable than the SparseBartels-Golub Method.

The Forrest-Tomlin Method

The Forrest-Tomlin Method

Bartels-Golub Method Forrest-Tomlin Methodd = U−1L−1b d = U−1

∏t RtL

−1bc = c ′N − c ′BU−1L−1N,{q|cq = min

t(ct)}

c = c ′N − c ′BU−1∏

t RtL−1N,

{q|cq = mint

(ct)}cq ≥ 0, d is optimal solution cq ≥ 0, d is optimal solutionw = U−1L−1Aq,{

p di

wi= min

t

(dt

wt

), wt > 0

} w = U−1∏

t RtL−1Aq,{

p di

wi= min

t

(dt

wt

), wt > 0



Update U−1 and L−1 Update U−1 creating a row factoras necessary. If there are toomany factors, completely refactorthe basis.

The Forrest-Tomlin Method CharacteristicsAdvantages:

I At most one row-eta matrix factor will occur for each iterationwhere an unpredictable number occurred before.

I The code can take advantage of such knowledge for predictingnecessary storage space and calculations.

I Fill-in should also be relatively slow, since fill-in can only occurwithin the spiked column.

Disadvantages:

I Sparse Bartels-Golub Method allowed LU-decomposition to pivot fornumerical stability, but Forrest-Tomlin Method makes no suchallowances.

I Therefore, severe calculation errors due to near-singular matrices aremore likely to occur.

Suhl-Suhl Method

This method is a modification of Forrest-Tomlin Method.

For More Detail

Leena M. Suhl, Uwe H. SuhlA fast LU update for linear programming.Annals of Operations Research 43(1993)33-47

Stiven S. MorganA Comparison of Simplex Method AlgorithmsUniversity of Florida, 1997

Vasek ChvatalLinear ProgrammingW.H. Freeman & Company (September 1983)

http://www.cise.ufl.edu/~davis/Morgan

http://users.encs.concordia.ca/~chvatal/

Thanks

Sparse Linear Algebra: LU Factorization - McMaster …cs777/presentations/LU...Once the L and U...

Documents

Transcript of Sparse Linear Algebra: LU Factorization - McMaster …cs777/presentations/LU...Once the L and U...