SVD:

Singular Value

Decomposition

2

Motivation

yUx

bLy

bLUx

LUA

A

matrixany :

bSSx

SSA

SSA

e-vectorsete set ofwith compl

A

11

111

1

matrixany :

bQQx

QQA

QQA

symmetricA

T

T

T

1

11

matrix :

bUVx

UVA

VUA

A

T

T

T

1

11

matrixany :

Clearly

the winner

Assume A full rank

3

Ideas Behind SVD

Goal: for Am×n

find orthonormal bases for C(AT) and C(A)

row space

Ax=0

column space

ATy=0

orthonormal basis in C(AT)

orthonormal basis in C(A)

A Rn Rm

There are many choices of

basis in C(AT) and C(A), but we

want the orthonormal ones

4

SVD (2X2)

2

2

2

2

1

1

1

1

21

21

21

,

:)(in imagesr want thei we

:)(in rsunit vecto

2 Assume

Av

Av

AvAv

Av

Av

T

uu

AvAvACAND

vvAC

r

: represent the length of images; hence non-negative

UAV

uuuuvvA2

1

21221121

I haven’t told

you how to find vi’s (p.9)

5

SVD 2x2 (cont)

TVUVUA

UAV

1

When A has complete set of e-vectors, we have

AS=S , A=SS1

but S in general is not orthogonal

Another diagonalization using 2 sets of orthogonal bases

When A is symmetric, we have

A=QQT

Compare

6

Why are orthonormal bases good?

( )-1=( )T

Implication:

Matrix inversion

Ax=b TT

T

UVVUA

VUA

111

system diagonal ... bUxV

bxVU

bAx

VUA

TT

T

T

7

More on U and V

TTTTT

TTTTT

UUUVVUAA

VVVUUVAA

2

2

2

1

2

2

2

1

Similarly,

V: eigenvectors of ATA U: eigenvectors of AAT

[Caution]: they (ui and vi) may differ in signs …

Find vi first, then use Avi to find ui

This is the key to solve

SVD

8

SVD: A=UVT

The singular values are the diagonal entries

of the matrix and are arranged in

descending order

The singular values are always real (non-

negative) numbers

If A is real matrix, U and V are also real

9

Example (2x2, full rank)

2121

2121

20

022

10

01

1

0,

1

0

2

0

0

1,

0

1

0

22

21

21,

21

21,

53

35

11

22

222

111

21

T

T

VUA

uAv

uAv

vvAA

A

STEPS: 1. Find e-vectors of ATA;

normalize the basis 2. Compute Avi, get i

If i 0, get ui

Else find ui from N(AT)

10

SVD Theory

If j=0, Avj=0vj is in N(A) The corresponding uj in N(AT)

[UTA=VT=0]

Else, vj in C(AT)

The corresponding uj in C(A)

#of nonzero j = rank

rjuAv

UAV

jjj ,,2,1,

11

11

11

2

1

00

010

21

12

5

1

11

22

2

1

5

100)(

1

1

2

10)(

101

2

5

1

1

1

2

1

11

22

1

1

2

1;

1

1:basis )(,1,

11

22

122212

22212

11

111

1

T

TTT

T

VU

uAuuAANuuu

vAvANvvv

uAv

vACrA

Example (2x2, rank deficient)

Can also be obtained from

e-vectors of ATA

12

Example (cont)

Problem: Sign and correspondence (if more than one ui for N(A)) matter?

11

11

2

1

00

010

21

12

5

1

11

22TVU

T

T

T

T

T

vuv

uu

v

vuuVUA

111

11

21

2

11

21

0

00

0

Bases of N(A) and N(AT) (u2 and v2 here) do not contribute the final result. The are computed to make U and V orthogonal.

13

Extend to Amxn

0

,,1,

1

1111

r

mrrnrr

iii

uuuuvvvvA

riuAv

Basis of N(A) Basis of N(AT)

T

mmmm

T

mm

TTT

mm

T

T

nnnn

T

nn

TTT

nn

T

T

nnnmmmnm

VUVUVUAA

VVVUVUAA

VUA

UAV

Dimension Check

14

Extend to Amxn (cont)

T

rrr

T

T

m

T

r

T

r

T

rr

r

mrrnrr

vuvu

v

v

v

v

uuA

uuuuvvvvA

1111

1

11

1

1111

00

0

Bases of N(A) and N(AT) do not contribute

They are useful only for nullspace solutions

Summation of r rank-one matrices!

15

110

121

011

:

,2,110

01132223332

AAV

UVArA

T

)(nullspace

1

1

1

3

1,0

1

0

1

2

1,1

1

2

1

6

1,3

33

22

11

v

v

v

1,1

1

2

1

3,1

1

6

23

3

3

6

1

22

12

11

Av

Av

3

1

3

1

3

1

2

1

2

1

2

1

6

2

6

1

3

1

2

1

6

1

3

1

6

2

3

1

2

1

6

1

01

3

11

11

2

1

110

011

1

3

11

11

2

10

110

011

C(AT) N(A) C(A)

16

21

12:

,2,

10

11

01

23332223

AAV

UVArA

T

1

1

2

1,1

1

1

2

1,3

22

11

v

v

ortherwiseor , ofvector -ein obtained )...( of basis 0

1,

1

0

1

2

1

3,

1

2

1

3

1

2

1

2

6

1

2

1

2

1

3

22

16

1

6

11

TT AAANAv

Av

Av

11

11

2

11

3

0

10

11

01

1

3

011

11

2

1

10

11

01

3

1

2

1

6

1

3

1

6

2

3

1

2

1

6

1

3

1

2

1

6

1

3

1

6

2

3

1

2

1

6

1

C(AT) C(A) N(AT)

It’s alright if we fill N(AT) with 0 if

we only care about C(AT) and C(A), the ones correspond to nonzero ’s

17

Summary

SVD chooses the right basis for the 4 subspaces

AV=U

v1…vr: orthonormal basis in Rn for C(AT)

vr+1…vn: N(A)

u1…ur: in Rm C(A)

ur+1…um: N(AT)

These bases are not only , but also Avi=iui

High points of Linear Algebra Dimension, rank, orthogonality, basis, diagonalization, …

18

SVD Applications

Using SVD in computation, rather than A, has the advantage of being more robust to numerical error

Many applications: Image compression

Solve Ax=b for all cases (unique, many, no solutions; least square solutions)

rank determination, matrix approximation, …

SVD usually found by iterative methods (see Numerical Recipe, Chap.2)

19

SVD and Ax=b (mn)

Check for existence of solution

existnot doessolution

,0but 0 If

ii

d

T

z

TT

d

dz

bUxVbxVU

20

Ax=b (inconsistent)

52

520

3

8

21

12

5

1

00

010

11

11

2

1

00

010

21

12

5

1

11

22

3

8,

11

22

2

1bU

z

z

VU

bA

T

T

No solution!

21

Ax=b (underdetermined)

21

21

2

2

2

2

0

22

11

11

2

1

0

22

0

520

4

8

21

12

5

1

00

010

11

11

2

1

00

010

21

12

5

1

11

22

4

8,

11

22

2

1

2

1

cxxx

Vzxz

z

bUz

z

VU

bA

nullparticularcomplete

particular

T

T

22

Pseudo Inverse

(Sec7.4, p.395)

The role of A:

Takes a vector vi from row space to iui in the

column space

The role of A-1 (if it exists):

Does the opposite: takes a vector ui from column

space to row space vi

ii

iii

iii

vuA

uAv

uAv

i

11

1

Fig 7.4, p.396

23

Pseudo Inverse (cont)

While A-1 may not exist, a matrix that takes ui back

to vi/i does exist. It is denoted as A+, the pseudo

inverse

A+: dimension n by m

T

mr

r

nr

T

mmmnnn

ii

i

i

uuuvvv

UVA

riuArivuA

11

1

1

1

for 0 and for 1

24

Pseudo Inverse and Ax=b

Full rank: A-1 exist; A+ is the same as A-1

Underdetermined case: many solutions, but will find the one with

the smallest magnitude |x|

Overdetermined case: find the solution that minimize the error

r=|Ax–b|, the least square solution

[proofs given below]

A panacea for Ax=b

bUVbAx

bAx

T

25

[Proofs from NR-1]

26

[Proofs from NR-2]

27

Ex: full rank

1

1

2

0

10

01

0

0

2121

2121

)/1(diag

2121

2121

20

022

10

01

2

0,

11

22

2

1

22

1

x

bUVxbxVUbAx

VUA

bA

T

i

T

T

28

Ex: under-determined

32

32

34

2

1

23

1

23

2

2

1

23

1

2

1

2

1

2

1

2

13

1

2

1

6

1

6

2

2

1

6

1

3

1

3

1

3

1

2

1

2

1

6

1

6

2

6

1

2

1

2

1

2

1

2

1

3

1

2

1

6

1

3

1

6

2

3

1

2

1

6

1

2

20

0

210

01

3

110

011

1

3

11

11

2

10

110

011

0

2

110

011

z

y

x

bUVbAx

xbAx

T

29

Under-determined (cont)

xbAx

Rcx

zy

yx

smallest with particular a gives

in line a lly,Geometrica

1

1

1

0

0

2

:Solution Complete

0

2

3

32

32

34

1

1

1

30

Ex: over-determined

T

T

UVA

VUA

UVArbA

11

11

2

11

3

0

10

11

01

,2,

1

2

1

,

10

11

01

3

1

2

1

6

1

3

1

6

2

3

1

2

1

6

1

23332223

Will show this need not be computed…

31

Over-determined (cont)

31

35

3

22

2

2

6

4

3

1

2

1

2

1

6

1

6

2

6

1

3

1

211

11

2

1

2111

11

2

1

1

2

1

0111

11

2

1

zyx

zyx

bUVx T

31

35

2

1

2

1

1

3

21

12

ˆ Compare

x

x

x

x

bAxAA TT

Same result!!

32

Ex: general case, no solution

1019

1019

101

51

101

51

5

1

5

2

52

1

52

1

5

1

5

2

10

1

2

1

2

1

3

8

3

8

0

0

3

8

00

0

11

11

2

1

00

010

21

12

5

1

11

22

3

8,

11

22

yx

yxy

x

bUVx

VU

bA

T

T

3

822

yx

yx

33

Matrix Approximation

nm

nm

vuvuvuA

AiA

imi

VUA

T

mmm

TT

i

i

T

ii

:saveOperation

numbers )(matrix onerank :save Storage

distanceEuclidean of sence in the ion toapproximat rank best the:

zero) tos' last setting(by of version rank the:

222111

making small ’s to zero and back substitute

(see next page for application in image compression)

34

Image Compression

As described in text p.352

For grey scale images: mn bytes

Only need to store r(m+n+1)

T

rrr

TT vuvuvuA

rSVD

222111

: termst significanmost the taking,After

r = 1,3,5,10,16 (no perceivable difference afterwards) Original

6464

Play with the demo program