Matrix factorization

Matrix Factorization

Lecture #7

EEE 574

Dr. Dan Tylavsky

© Copyright 1999 Daniel Tylavsky

Matrix Factorization�

Let’s look at solving Ax=b

–

A, b are dense.

�

Two possible (general approaches):

–

Indirect Methods - methods which asymptotically approach (but never reach) the true solution as the number of steps increases.

•

Ex: Point Jacobi, Gauss-Siedel, Successive Over relaxation, Multigrid


Matrix Factorization–

Direct Methods - methods that yield the theoretically exact result in a fixed number of steps. (Assuming an infinitely precise computing engine.)

•

Ex: Elimination Methods

–

Gauss Elimination

–

Gauss-Jordan Elimination

–

Cholesky Factorization (Numerically Symmetric)

–

LU or LDU factorization (a.k.a., Product form of the Inverse),(Variations Include Crout, Doolittle, Banachiewicz)

•

Orthogonalization Methods

–

QR factorization

–

Given Rotations


Matrix Factorization•

Conjugate Gradient Methods - Exact solution in # of steps equal to the number of different eigenvalues.

–

Various Variations

�

Advantages of Elimination Methods

–

No convergence criteria (however, pivoting in general non-positive definite case.)

–

Factors can be used repeatedly.

–

Factors can be ‘easily’ modified to accommodate matrix/network changes.

–

Partial factorized matrices give linear network equivalents.



Definitions

•

Symmetric: A=AT

•

Positive Definite: xTAx>0, (A>0 is short hand notation)

•

Diagonally Dominant:

∑≠

≥ij

ijii aa ,with inequality for at least one i.

• Properly Diagonally Dominant: ∑≠

>ij

ijii aa for all i.

• Rank: Order of the largest sub-matrix with nonzero determinant.



Properties

•

Symmetric pos. def. (A>0) matrix has all positive real eigenvalues.

•

If A>0, then Det(A)=|A|>0.

•

A rank 1 matrix must have the outer-product form, xyT.

• Symmetric A>0 has a unique Cholesky factorization. A=UTU, A=U’ TDU’

• If A is properly diagonally dominant and all of its diagonal elements are positive, then A is positive definite.

[ ]642,

3

2

1

=

= Tyx

=

18126

1284

642Txy


Matrix Factorization– Factorizing a non-symmetric matrix.

=

=11

11

321

3333231

2232221

1131211

~Ac

b

aaaa

aaaa

aaaa

aaaa

AT

nnnnn

n

n

n

α

– I claim A can be expanded in the following form:

12

11

1

21

1

0

10

010UAL

I

b

AIcA

T

=

= α

α


Matrix Factorization– To find the value of A2, multiply.

12

11

1

21

1

0

10

010UAL

I

b

AIcA

T

=

= α

α

=

+=

=

11

11

1

1121

11

1

1

21

1~

0

10

Ac

bbcAc

b

I

b

AcA

TT

TTα

α

αα

α

– Therefore:

11

112

~AbcA

T=+∴ α 1

1112

~α

TbcAA −=⇒

Outer product term of first row & first column (absent. diag. element)


Matrix Factorization– Do this repeatedly until we have the product form of A.

1

22

22112

112

1 ~0

01

0

01U

Ac

bLUA

LUALA T

TT

=

== α

=

=

I

b

AIcAcbA

T

T

T

T

T

T

T

T

00

0

10

01

000

0

10

01

00

0

0

01

~0

0

0

01

2

2

32

2

22

222

ααα

23

2 UAL=


Matrix Factorization– To find A3, multiply the result.

23

22

2

32

22

00

0

10

01

000

0

10

01

00

0

0

01UAL

I

b

AIc

AT

T

T

T

T

T

=

= αα

=

I

b

Ac

T

T

T

T

00

0

10

01

00

0

0

01

2

2

32

2 αα

+=

2

2232

22

0

0

0

01

α

αT

T

T

bcAc

b

22

223

~AbcA

T=+∴ α 2

2223

~α

TbcAA −=⇒– We can write:

123

21 UUALLA =


Matrix Factorization– Continuing on yields:

121121 UUUALLLA nn

n −−=

– where:

n

nnn

n L

a

A =

=10

0

1

121121 UUULLLLA nnn −−=

– We can write:



jij

i

cc

LL

ji

j

i

ji <←←

= ,

1

1

1

αα

–

Elementary-Matrix-Multiplication-Superposition Principal.

jij

i

cc

LL

j

j

i

iji <←

←

= ,

1

1

1

1

1

1

1

αα

i j i j



This implies I can write:

L

c

c

cLLLL

n

nn =

=−

α

αα

α

3

2

1

3

2

1

121



Taking the transpose of the elementary-matrix- multiplication-superposition principal allows me to write:

Ub

b

b

UUUUT

T

T

nn =

=−−

1

1

31

1

1

3

2

2

1

1

1221

α

α

α

–

Hence we can write:

LUA =



Let’s work though an example:

10.000 -1.000 -2.000 -3.000-1.000 12.100 -2.200 -2.700

A= -2.000 -2.200 15.880 -6.300-3.000 -2.700 -6.300 23.400

–Divide Top Row by a11.

– Compute

– Compute

1

1112

~α

TbcAA −=

2

2223

~α

TbcAA −=

–Divide 2nd Row by α2.

10.000 0.000 0.000 0.000 1.000 -0.100 -0.200 -0.300-1.000 12.100 -2.200 -2.700 0 1 0 0

A= -2.000 -2.200 15.880 -6.300 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0.000 0.000 0.000 1.000 -0.100 -0.200 -0.300-1.000 12.100 -2.200 -2.700 0 1 0 0

A= -2.000 -2.200 15.880 -6.300 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0.000 0.000 0.000 1.000 -0.100 -0.200 -0.300-1.000 12.000 -2.200 -2.700 0 1 0 0

A= -2.000 -2.200 15.880 -6.300 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0.000 0.000 0.000 1.000 -0.100 -0.200 -0.300-1.000 12.000 -2.200 -2.700 0 1 0 0

A= -2.000 -2.200 15.880 -6.300 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0.000 0.000 0.000 1.000 -0.100 -0.200 -0.300-1.000 12.000 -2.400 -3.000 0 1 0 0

A= -2.000 -2.200 15.880 -6.300 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0.000 0.000 0.000 1.000 -0.100 -0.200 -0.300-1.000 12.000 -2.400 -3.000 0 1 0 0

A= -2.000 -2.200 15.880 -6.300 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0.000 0.000 0.000 1.000 -0.100 -0.200 -0.300-1.000 12.000 -2.400 -3.000 0 1 0 0

A= -2.000 -2.400 15.480 -6.900 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0.000 0.000 0.000 1.000 -0.100 -0.200 -0.300-1.000 12.000 -2.400 -3.000 0 1 0 0

A= -2.000 -2.400 15.480 -6.900 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0 0 0 1.000 -0.100 -0.200 -0.300-1.000 12.000 -2.400 -3.400 0 1 0 0

A= -2.000 -2.400 15.480 -6.900 * 0 0 1 0-3.000 -3.000 -6.900 22.500 0 0 0 1

10.000 0 0 0 1.000 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.400 15.480 -6.900 * 0 0 1 0-3.000 -3.000 -6.900 22.500 0 0 0 1

10.000 0 0 0 1.000 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.400 15.480 -6.900 * 0 0 1 0-3.000 -3.000 -6.900 22.500 0 0 0 1

10.000 0 0 0 1.000 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.400 15.000 -7.500 * 0 0 1 0-3.000 -3.000 -7.500 21.750 0 0 0 1

–Divide 3rd Row by α3.

10.000 0 0 0 1.000 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.400 15.000 0 * 0 0 1 -0.500-3.000 -3.000 -7.500 21.750 0 0 0 1

– Compute3

3334

~α

TbcAA −=

10.000 0 0 0 1.000 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.400 15.000 0 * 0 0 1 -0.500-3.000 -3.000 -7.500 21.750 0 0 0 1

10.000 0 0 0 1.000 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.400 15.000 0 * 0 0 1 -0.500-3.000 -3.000 -7.500 18.000 0 0 0 1

10.000 0 0 0 1.000 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.400 15.000 0 * 0 0 1 -0.500-3.000 -3.000 -7.500 18.000 0 0 0 1


Matrix Factorization� Individual Problem: Verify that previous answer is correct.

10.000 0 0 0 1.000 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.400 15.000 0 * 0 0 1 -0.500-3.000 -3.000 -7.500 18.000 0 0 0 1

� Note that result isn’t symmetrical.

� Individual Problem: Factor A=LU for:20.000 -20.000 -8.000 -8.000-20.000 40.000 3.000 3.000

A= -8.000 3.000 14.450 -5.550-8.000 3.000 -5.550 24.450


Matrix Factorization� To make this symmetric use the L’DL’T form.

– Let

10.000 0 0 0 1.000 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.400 15.000 0 * 0 0 1 -0.500-3.000 -3.000 -7.500 18.000 0 0 0 1

=

44

33

22

11

l

l

l

l

D

– Write:TDLLDULDA ''1 == −

1 0 0 0 10 0 0 0 1.000 -0.100 -0.200 -0.300-0.100 1 0 0 0 12 0 0 0 1 -0.200 -0.250

A= -0.200 -0.200 1 0 * 0 0 15 0 * 0 0 1 -0.500-0.300 -0.250 -0.500 1 0 0 0 18 0 0 0 1

– Division by diag. of row & col. Performed @ same time.


Matrix Factorization� Individual Problem: Modify LU factors of A:

� To get L’DL’T:

20.000 -20.000 -8.000 -8.000-20.000 40.000 3.000 3.000

A= -8.000 3.000 14.450 -5.550-8.000 3.000 -5.550 24.450



10.000 0 0 0 1.000 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.400 15.000 0 * 0 0 1 -0.500-3.000 -3.000 -7.500 18.000 0 0 0 1

� Data structure for storing LU factors?

� For LU factors could use LC-D-UR/X:

Pos: 0 1 2 3 4 5 6 7 8 9 10 11 12DiagL(k) 10.0 12 15 18

U(k): -0.1 -0.2 -0.3 -0.2 -0.3 -0.5L(k): -1.0 -2.0 -3.0 -2.4 -3.0 -7.5

Indx(k): 2 3 4 3 4 4ERP(k): 0 3 5 6



� For L’DL’T factors could use CR(L)-D/X:

Pos: 0 1 2 3 4 5 6 7 8 9 10 11 12Diag(k) 10.0 12 15 18

L(k): -0.1 -0.2 -0.3 -0.2 -0.3 -0.5RIndx(k): 2 3 4 3 4 4

ERP(k): 0 3 5 6

1 0 0 0 10 0 0 0 1.000 -0.100 -0.200 -0.300-0.100 1 0 0 0 12 0 0 0 1 -0.200 -0.250

A= -0.200 -0.200 1 0 * 0 0 15 0 * 0 0 1 -0.500-0.300 -0.250 -0.500 1 0 0 0 18 0 0 0 1



10.000 -1.000 -2.000 -3.000-1.000 12.100 -2.200 -2.700

A= -2.000 -2.200 15.880 -6.300-3.000 -2.700 -6.300 23.400–

Divide Top Row by a11.

– Compute for 2nd row.1

1112

~α

TbcAA −=–

Divide 2nd Row by α2.

10.000 0 0 0 1 -0.100 -0.200 -0.300-1.000 12.100 -2.200 -2.700 0 1 0 0

A= -2.000 -2.200 15.880 -6.300 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0 0 0 1 -0.100 -0.200 -0.300-1.000 12.100 -2.200 -2.700 0 1 0 0

A= -2.000 -2.200 15.880 -6.300 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0 0 0 1 -0.100 -0.200 -0.300-1.000 12.000 -2.200 -2.700 0 1 0 0

A= -2.000 -2.200 15.880 -6.300 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0 0 0 1 -0.100 -0.200 -0.300-1.000 12.000 -2.200 -2.700 0 1 0 0

A= -2.000 -2.200 15.880 -6.300 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0 0 0 1 -0.100 -0.200 -0.300-1.000 12.000 -2.400 -3.000 0 1 0 0

A= -2.000 -2.200 15.880 -6.300 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

–

If we store & work through A by rows, factorization is done in a slightly different way:

– Compute and for 3rd row.2

2223

~α

TbcAA −=1

1112

~α

TbcAA −=

10.000 0 0 0 1 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.200 15.880 -6.300 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0 0 0 1 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.200 15.880 -6.300 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0 0 0 1 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.400 15.880 -6.300 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0 0 0 1 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.400 15.880 -6.300 * 0 0 1 0

-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0 0 0 1 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.400 15.000 -6.300 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0 0 0 1 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.400 15.000 -6.300 * 0 0 1 0

-3.000 -2.700 -6.300 23.400 0 0 0 1

10.000 0 0 0 1 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.400 15.000 -7.500 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1


Matrix Factorization� Individual Problem: Finish the factorization.

10.000 0 0 0 1 -0.100 -0.200 -0.300-1.000 12.000 0 0 0 1 -0.200 -0.250

A= -2.000 -2.400 15.000 -7.500 * 0 0 1 0-3.000 -2.700 -6.300 23.400 0 0 0 1


Matrix Factorization– Cholesky Factorization (Symmetric A=LLT.)

=

=11

11

321

3332313

2232212

1131211

~Ab

b

aaaa

aaaa

aaaa

aaaa

AT

nnnnn

n

n

n

α

T

T

LAL

I

b

AIbA 12

11

11

21

1

1

00

010=

= α

α

α

α– A can be expanded in the following form:

Etc.

– Not used. Calculating square-roots is expensive.

The End

Matrix factorization

Education

Transcript of Matrix factorization