L14: Orthogonal Matching Pursuit & Lasso

and Compressed Sensing

Je↵ M. Phillips

February 26, 2020

Midterm• No computers ,

calculators, planes

• I ' ' cheat she - t" ( both sides )

• Understand core definitions3 Qeesttxon C sub parts)

• Clustering-

Assessment - bas .ee

- HACa

'

k - grams• Similarityo Distance a Streaming

.Mitha shrug

NG

n Jace card

Ridge & Lasso Regression-

laps IcyX EIR

'd

get" X - ft ;D

God : RR : go=arsmi "

II. ( yi- 24 ;D )2tsll*R

a end " '

if rest?

Lasso ids = argentinian II. Cgi - Gord 't shake

-

I = Inseminated:( gi - 2x ;D } sit .

Hallie

#kill !

* = LIFE , II. Csi - LAW set . Hallett

equivalent tf chooses s → If

so Go =L : I RR )Let t= 1108115

or 29=27 C Lasso )

It = I n II Csi - Hird set .

lkh÷÷÷i÷÷÷÷:* .

Lasso Illustration

Find ↵⇤= argmin↵2Rd kX↵ � yk2 + sk↵k1

k↵k2 = t

k↵k1 = t

t↵�

t with ridge regression

↵� with OLS

↵�t with lasso

Hall lie

←

:

Matching Pursuit (MP)

Find ↵⇤= argmin↵2Rd kX↵ � yk2 + sk↵k1

Forward Subset Selection:

Matching Pursuit

Set r = y ; ↵ = 0.for i = 1 to t doSet Xj = argmaxXj02X |hr ,Xj 0i|.Set ↵j = argmin↵ kr � Xj↵k + s|↵|.Set r = r � Xj↵j .

Return ↵.

%. .

es

°

Orthogonal Matching Pursuit (OMP)

Find ↵⇤= argmin↵2Rd kX↵ � yk2 + sk↵k22

Forward Subset Selection:

Orthogonal Matching Pursuit

Set r = y ; ↵ = 0.for i = 1 to t doSet Xj = argmaxXj02X |hr ,Xj 0i|.Set ↵ = argmin↵ kr � [X1;X2; . . . ;Xj ]↵k + sk↵k22.Set r = r � Xj↵j . (Update using other ↵j 0 for j 0 < j)

Return ↵.

Ridge

full !! efficient'

a

Lasso Illustration

Find ↵⇤= argmin↵2Rd kX↵ � yk2 + sk↵k1

k↵k2 = t

k↵k1 = t

t↵�

t with ridge regression

↵� with OLS

↵�t with lasso

( east Angle Regression

µMP

i nst

¥D\÷*a. . . .¥i.on ,

Sparse Sensing

ST = [0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0 1 0 0 0 1 0 0 1 0 0]

xTi = [-1 0 1 0 1 1 -1 1 0 -1 0 0 1 -1 -1 1 0 1 0 1 -1 -1 -1 0 1 0 0 -1 0 1 0 0]

yi = hS , xi i= 0+0+0+0+1+0+0+0+0+0+0+0+0+0+0+1+0+0+0+1-1+0-1+0+0+0+0+0+0+1+0+0

= 2

what if d > n

but k ? n

keenLos I ( unknown )

ksnpasse C

-44279galbiddies ,

ord

"" " " "

I:*.

.

. " "

Matching Pursuit (OMP)

Matching Pursuit

Set r = y .for i = 1 to t doSet Xj = argmaxXj02X |hr ,Xj 0i|.Set �j = argmin� kr � Xj�k.Set r = r � Xj�j .

Return S where sj = �j (or 0).

OMP Example

signal: S = [0, 0, 1, 0, 0, 1, 0, 0, 1, 0]

measurement: X =

2

666664

0 1 1 �1 �1 0 �1 0 �1 0�1 �1 0 1 �1 0 0 �1 0 11 �1 1 �1 0 �1 1 1 0 01 0 �1 0 0 1 �1 �1 1 1�1 0 0 0 1 0 1 0 1 �10 0 �1 �1 �1 0 �1 1 �1 0

3

777775

observation: y = XST = [0, 0, 0, 1, 1,�2]T000

⑧

r,

= £ I,

o,

0, 0,0 ,

- I ]