Singular Value Decompositions

A men matrix

Aim Find orthonormal bases Bc R and Cc IR such that

Apic is as simple as possibleA

Matrix of Ta with respect to B and C

Important Observation AtAT ATA'T ATA ATA symmetric

nxn matrix

AtA symmetric ATA orthogonally diagonalizable

Let B Ii In C 112 an orthogonal basis at eigenvectors A ATAeigenvalue

ATA y Tiki 7W all i

Lit ATA z Litt y recall Hei11 1 Kitui l

A I care XiHA If X T 0 For all i

Perhaps after reordering assume 7 3727 7 Tu 0

Definition The singular values of A are the numbers

5 7 027 70 70 where T TX

key Fact c HA till

Theneur Assume 5 Tr F O and Gre Ju oThen Rank A r and Are A 3 is an orthogonalbasis For Col A

Range TaProof he 4 C 112 a basis Ay Ayn spans Cska

HA 11 ti Aei e Ei o

A Yi A Ir spans Col A and all elements are non zero

Let i jA I A I Aei TAej E'T ATA Ij VIT Ij

Xj Iitej Xj Ei Yj 0

Ei Ij orthogonal it i j

A Yi Ayer is orthogonal set with non zero vectors L I

AI AI is an orthogonal basis For ICA

Rank A r

Example

a IF E a IF E

p.ioa a

Choose Yi Ei Iz 13 13T T TAI 4 y AA Er Iz A AI E 4 I 0 are eigenvalues

2 I 0 are singular values

Ay AE Free l i basis For GILA

obvious in this case as I Ei Iz Ez

t HEYis y y.AE FAve

2

II T T maps the

hollow sphere to this

solid ellipserestricted to

sphereA radius

1

Important General Fact center ein IR

Max HAIK where HE11 1 Max OT Tn 5

Max 11Hell where 11111 d do

I restricted tosphere A radius dcenter e in R

M in HAIK where HE11 1 Mino on on

Min 11Hell where 11111 d don

Back to general case

let Ii Aei Ati Fm sie

Key Facts

y I Iir is an orthonormal basis Fu Colla

Y Aki Ti Ii Tor sie r

Observation we can extend I Iir is an orthonormal basis

for all 112 as follows

a Find basis for NullAT Colla t

b Apply Gram Schmidt and normalize to get orthonormal basisErtl In must have size m as

dim call A I mdim Colla fc Take union to give ki Er art Em an orthonormal

basis for IR

Singular Value Decompositions

Let It be an mxn matrix together with

413 y In C1R orthonormalbasis at eigenvectors et ATA with singular valueszero

01 3 02 3 Jr a S u CATA Ii Tikiz C I Em C Rm orthonormal basis with AI o yFor all I E n

Ap c AI c CAIL Aem AI cAMatrix

associatedto

y withrespectto

o le Lorenc E e cBand C

µ 0 JT often denoted ZO 0

Important Consequence If U y Lm ki EnT T u

i'Av no o TITEL F v

O O

A

g v u oro

O O

7Called a singular value decomposition

of A

Remarks

n TA m

R s R

I I AI nonzero

eOT di r

tio Imma

I Ir Yrt In ki Er _ur i Im

1CAT Nuh A Colla NU AT

Overview at Finding SVD i

1 Orthogonally diagonalize ATA giving orthonormalbasis I In

z Reorder so that eigenvalues are decreasing 7 77 7 In onon zero zero

Take square roots to give T za 70 707 7 En

iDefine Ii Aki Fm I sie r

Extend y yr to orthonormal basis ki Ir 4mi i Em

6 U Lee EunV y Lin A U I v'tE no o

s.v.isO 0

Terminology ki Eun left singular vectors at A

er yrs right singular vectors at A

SVD havemany important applications

Example Image Processing

Imagine we have a greyscale image such as the following

512 512

resolution

We can encode this data with a 512 512 matrix A

where each entry represents the brightness at the corresponding

pixel

SVD A U I v't

For anylek E 512 let II

And Ar U I zvT

It K E Ronk A Rank Ate k

Important Fact

A is a best possible rank k approximation of A

Pentries are

as

close as possible

Ak E E VI x 0 212 t t 5k IFT

Need 12 2k 512 numbers todetermine

Each Ak corresponds to a compressed version of original

image A

This has all sorts of applications in A.I cognative science

and machine learning not to mention Letting you stare more

pictures on your phone