S 6140

Homework 3 Solutions

Problem 1

Q1 e

Here d 2 6 3 and m 3 and

and we want to find Ae 113 such that

main119 Axl'll

We let OIOI05 e Bt be the

rows of A

A IEE eye

Then by the definition of the

Euclidean norm of a vector

119 Ax 11 É 19 A x

ji Gtxlil

Therefore We have

min119 Ax 11 mging É É ji Gtx

minsogo ÉÉ I gtying

É moi I y gtxcity

Here we have used the fact that

min

902,0191 1102 1,103

MEILEEmin Lalo minf.coOz

when flo so for all the Os

Let

M's Eff Elfand

x É JemmaKnit

We can therefore Solve 3 separate

linear regression problems

Note that multiplying by E doesn't changethe solution

L R1

me I E15 Ex mi I 11 XE K

LR 2

me I É y ofx min I11 XE K

LR 3

ng I E14 Osx min I11 s Xo It

The solutions are

q xix Xt 51

q Xix Xt314

O Xix XY 1Recalling that Q Oz O are the rows of Awe get

413 113

A 1413 4130 A

Q I b First Method

As in the previous case

We let

a

foOE

io

Then observe that

mix II 11y A x y

gig I I y a x gf

1,5 I 14 often

If we now use the notationgiventhen we see that

mix É II y A x if

117,545,14 XO

where Y is the J th row of Y

We then solve k separate linear Regression

problems

Oj XTXT TTY

Remark Dimensions

axe IN'd XX eRad Xx J e add

XTY E Rd

Then Oj E Rd

We then form A with found rows

Note now that

get FY Xix t

YTX Tx

where we used Ex FLEXI Xix

Therefore by stacking the Os as the

rows of A we obtain

A I E ETEJ

En Tata

Note that indeed A is kid

Q I b second Method

Recall that for a matrix Me 121

the Frobenius norm is

M KEITT TETEwhere Mi be the i th column of M

Note moreover that

IMI M MIF

where 2 A B7 Ez ATB is the

Frobenius inner product between A Bendit

Then notice that the loss can be written as

f A É 119 A X II HY AXT I

Therefore for Hedwe have

J At H HY AXT HXTII

Y AXT HXT Y AXY HXT

II Y AXTIE

2 HXT Y AXT It 11HXTIE

L A 2 HXT Y AXT OCHHIIIF

L A 2 H Y AXTXI OCHHII

where we used ABT CE LA CB

Then it follows that

flat H fCA t 2 H AXt Y X z t O AHIR

Recalling the definition of gradientwe find that

DJ A 2 AXT Y X

setting the gradient to zero

of A o

we obtain

AKTX Y X

which givesA IE CI 5

for rank d X

Q IC

given a new x once obtained A

we predict

Y A x

but since

A IE I 5

Then

y TIE III 5 x G

Note that if we let v be the vector

K E EET x

Then by equation G

Y V I y y

We can then conclude that

is is the span of the columns

of Y

Problem 2

za z of É Eze Éiffe

612 61272 612 1 612

2b fco É Y logis 11 Yi log l T

Note that if Zi O X then 51 612

Therefore we have

gZi ti

ffI 612 1 612 YiLi Ji

81,1 81 Yo Jill 5 Xi

Ofco is a vector

The j th entry of Of10 is

gfcoBy the previous results

go flo É YilayJi Ci Si log l T

if f g t i g Ill Ji Xiii

I I 5 Yi Xi

Therefore

go f o I I 5 Yi Xi

or in Vectorform

ofco É II Yi Xi

Q2 C

If M É XiXt then for any Z

M z É Xi XYZso that

2TMZ II XYZ

É xiz 20 because any xiztzo

QIH is a dxol symmetric matrix

The entry j k of this matrix is

Hir fgfqfloffgfg.fm Hrj

Therefore

Hip gaff 15 Yi Xi

fat Xi

ZI Jill Ji Xi Xi k

Having used the result

Note now that this recall problem Q2c

Indeed observe that the entry j K

of ZI XiXi is

É XiXi Xi is Xi k

Therefore we recognize thatsince

H.jp É Ii l Ji Xi Xi kThen

H É bill Si Xi Xt

Next observe that for any Z

Z Hz É Ii l T Xi z

Moreover note that Oc p at therefore

p 1 p o for all the p

Therefor is l Y o for all the i

In Conclusion Till 5 Xi 2520 for all the iand therefore THE 20 for all Z